[Openerp-community] Fw: Re: odoo with a daily 50000 orders volume

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Openerp-community] Fw: Re: odoo with a daily 50000 orders volume

LIN Yu



 Original Message 
Sender: lin.yu<[hidden email]>
Recipient: Raphael Valyi<[hidden email]>
Cc: Eric Caudal<[hidden email]>
Date: Monday, Aug 25, 2014 11:22
Subject: Re: [Openerp-community] odoo with a daily 50000 orders volume

Hello Raphael,


Thanks for your fast response. They are really helpful.


We have worked a daily 5000 order lines, 30000 moves project before with another company in V7. After several months, we had terrible performance issues as well. Especially in stock moves and procurements. 


As you mentioned on V8 stock scalability issue is gone. I doubt that still. The new stock level is calculated based on the stock quant history, which will be millions of lines in our case. And what’s more, each stock operation needs to operate on stock quant objects (aplication level + database level), besides the old objects in V7.

Personally I think we need an archive function to save old stock data in another table, redefine the stock level function based on the archive result (inventory line or another seperate table).


For the workflow, I do not know how to deal with it yet. Maybe we should get rid of workflow on each stock operations.


With the volume increases, it requires the system to lighten the resource of every operations . 

It would be great to have a proof that odoo / openerp can work  on that level volume.

Look forward to receiving more feedbacks from the community.


BR,
 Original Message 
Sender: Raphael Valyi<[hidden email]>
Recipient: lin.yu<[hidden email]>
Cc: OE Community [hidden email]<[hidden email]>
Date: Monday, Aug 25, 2014 10:43
Subject: Re: [Openerp-community] odoo with a daily 50000 orders volume

Hello Lin,

we worked indirectly on a case in the USA with 1/10 of the numbers you mentioned. And we had terrible issues. Just to be clear we were called as firemen and didn't had a chance to study things upfront on that project. Well basically, 2 things became extremely slow: workflows because tables were huge and stock moves (that was on 6.1) because millions of move to crawl to compute any stock level.

On v8, there is good hope that the stock scalability issue is gone. As for the workflows I'm afraid I'll probably still have issues: all workflows use the same tables and open workflow instances of sale, stock, account accumulate. My advice is: if it's on a version prior to v8, forget it because of the way the stock moves work. If it's on v8, well in any case plan some very serious optimizations and testing to see feasibility.

Number of concurrent users isn't too hard to tackle by adding more cores now that we have process based parallelism, but as for very large tables, well you'll have to clean them up often, you'll have tweak indexes, probably to bypass some computations and use some custom SQL, add caching...
I never heard about sharding in OpenERP/Odoo so I guess nobody ever did sharding, so don't assume it would be easy like in frameworks where it's being done...

As for the company we worked for, well, after 18 months they ended up dropping OpenERP for SAP. Probably putting way more money on it than trying again on v8 and paying a lot of engineering to get the things optimized, but still it's the decision they took and scalability was one of the main reasons and that was 1/10th of the figures you mentioned. So don't take that challenge lightly and good luck!

Regards.


-- 
Raphaël Valyi
Founder and consultant
+55 21 3942-2434




On Sun, Aug 24, 2014 at 10:21 PM, lin.yu <[hidden email]> wrote:

Hello Community,


We have a potential lead which has a volume of daily 20000 sale orders and 50000 sale order lines from Magento, which will probably result in a daily 100000 or 200000 stock moves, 3000000 stock moves per month.

I would like to ask that has anyone experience on Odoo/OpenERP with that daily volume? How about the performance after one month?


I would like also to ask if you have any module on 

* archive stock move , sales order, and other objects

* sales forecasting


BR,


_______________________________________________
Mailing list: https://launchpad.net/~openerp-community
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-community
More help   : https://help.launchpad.net/ListHelp



_______________________________________________
Mailing list: https://launchpad.net/~openerp-community
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-community
More help   : https://help.launchpad.net/ListHelp
Reply | Threaded
Open this post in threaded view
|

Re: [Openerp-community] Fw: Re: odoo with a daily 50000 orders volume

Grahame Jordan
Hi,

The problem with calculating stock levels can be overcome by adding a column that stores the current level.
Evey stock move then can modify the stock level column.
I would consider using SQL directly to do this task to avoid ORM slowness
Much less work, maybe up to 80x faster, at least in 6.1.

Grahame

On 25/08/14 13:42, lin.yu wrote:



 Original Message 
Sender: lin.yu[hidden email]
Recipient: Raphael Valyi[hidden email]
Cc: Eric Caudal[hidden email]
Date: Monday, Aug 25, 2014 11:22
Subject: Re: [Openerp-community] odoo with a daily 50000 orders volume

Hello Raphael,


Thanks for your fast response. They are really helpful.


We have worked a daily 5000 order lines, 30000 moves project before with another company in V7. After several months, we had terrible performance issues as well. Especially in stock moves and procurements. 


As you mentioned on V8 stock scalability issue is gone. I doubt that still. The new stock level is calculated based on the stock quant history, which will be millions of lines in our case. And what’s more, each stock operation needs to operate on stock quant objects (aplication level + database level), besides the old objects in V7.

Personally I think we need an archive function to save old stock data in another table, redefine the stock level function based on the archive result (inventory line or another seperate table).


For the workflow, I do not know how to deal with it yet. Maybe we should get rid of workflow on each stock operations.


With the volume increases, it requires the system to lighten the resource of every operations . 

It would be great to have a proof that odoo / openerp can work  on that level volume.

Look forward to receiving more feedbacks from the community.


BR,
 Original Message 
Sender: Raphael Valyi<[hidden email]>
Recipient: lin.yu<[hidden email]>
Cc: OE Community [hidden email]<[hidden email]>
Date: Monday, Aug 25, 2014 10:43
Subject: Re: [Openerp-community] odoo with a daily 50000 orders volume

Hello Lin,

we worked indirectly on a case in the USA with 1/10 of the numbers you mentioned. And we had terrible issues. Just to be clear we were called as firemen and didn't had a chance to study things upfront on that project. Well basically, 2 things became extremely slow: workflows because tables were huge and stock moves (that was on 6.1) because millions of move to crawl to compute any stock level.

On v8, there is good hope that the stock scalability issue is gone. As for the workflows I'm afraid I'll probably still have issues: all workflows use the same tables and open workflow instances of sale, stock, account accumulate. My advice is: if it's on a version prior to v8, forget it because of the way the stock moves work. If it's on v8, well in any case plan some very serious optimizations and testing to see feasibility.

Number of concurrent users isn't too hard to tackle by adding more cores now that we have process based parallelism, but as for very large tables, well you'll have to clean them up often, you'll have tweak indexes, probably to bypass some computations and use some custom SQL, add caching...
I never heard about sharding in OpenERP/Odoo so I guess nobody ever did sharding, so don't assume it would be easy like in frameworks where it's being done...

As for the company we worked for, well, after 18 months they ended up dropping OpenERP for SAP. Probably putting way more money on it than trying again on v8 and paying a lot of engineering to get the things optimized, but still it's the decision they took and scalability was one of the main reasons and that was 1/10th of the figures you mentioned. So don't take that challenge lightly and good luck!

Regards.


-- 
Raphaël Valyi
Founder and consultant
+55 21 3942-2434




On Sun, Aug 24, 2014 at 10:21 PM, lin.yu <[hidden email]> wrote:

Hello Community,


We have a potential lead which has a volume of daily 20000 sale orders and 50000 sale order lines from Magento, which will probably result in a daily 100000 or 200000 stock moves, 3000000 stock moves per month.

I would like to ask that has anyone experience on Odoo/OpenERP with that daily volume? How about the performance after one month?


I would like also to ask if you have any module on 

* archive stock move , sales order, and other objects

* sales forecasting


BR,


_______________________________________________
Mailing list: https://launchpad.net/~openerp-community
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-community
More help   : https://help.launchpad.net/ListHelp




_______________________________________________
Mailing list: https://launchpad.net/~openerp-community
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-community
More help   : https://help.launchpad.net/ListHelp


--
Glass Expansion
15 Batman Street
West Melbourne
Victoria 3003
AUSTRALIA
T: +61 3 9320 1111
F: +61 3 9320 1112
www.geicp.com


Click here to check out our new TruFlo Sample Monitor


404 Not Found

Not Found

The requested URL /site/dbimages/TruFlo.jpg was not found on this server.


Apache/2.2.22 (Debian) Server at geicp.com Port 80

_______________________________________________
Mailing list: https://launchpad.net/~openerp-community
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-community
More help   : https://help.launchpad.net/ListHelp
Reply | Threaded
Open this post in threaded view
|

Re: [Openerp-community] Fw: Re: odoo with a daily 50000 orders volume

Enapps :: Vadim Chobanu
In reply to this post by LIN Yu

Yes, workflow needs removing if this is to work for larger setups. We rebuilt the whole logic by replacing it with regular functions, optimising the code and the result was that what used to take 30 seconds to confirm a larger order+allocating stock now is instant and no fear of unknowns.
As for archiving its also a good idea, it can be implemented without moving the records out of the table though, just mark them as archived and when doing sql to compute the availability/pricing look at just un-archived records.

But it was a painful experience of getting this done.

 

 

Vadim Chobanu

Director

enapps logo email

 

T: +44 020 8090 9222 ext 1005

M: +44 079 3923 0959

E: [hidden email]

W: www.enapps.co.uk

-------------------------------------------------------------------------
Information in this email is confidential and may be privileged.
It is intended for the addressee only. If you have received it in error,
please notify the sender immediately and delete it from your system.
You should not otherwise copy it, retransmit it or use or disclose its
contents to anyone.
Thank you for your co-operation.
-------------------------------------------------------------------------

Address: 88 Wood Street; 10th floor; London EC2V 7RS

Registered in England No: 07746264

Reg. VAT No. GB 124 1001 86

 

 

 

On 25 Aug 2014 04:59, "lin.yu" <[hidden email]> wrote:

 

 

 Original Message 

Sender: lin.yu<[hidden email]>

Recipient: Raphael Valyi<[hidden email]>

Cc: Eric Caudal<[hidden email]>

Date: Monday, Aug 25, 2014 11:22

Subject: Re: [Openerp-community] odoo with a daily 50000 orders volume

 

Hello Raphael,

 

Thanks for your fast response. They are really helpful.

 

We have worked a daily 5000 order lines, 30000 moves project before with another company in V7. After several months, we had terrible performance issues as well. Especially in stock moves and procurements. 

 

As you mentioned on V8 stock scalability issue is gone. I doubt that still. The new stock level is calculated based on the stock quant history, which will be millions of lines in our case. And what’s more, each stock operation needs to operate on stock quant objects (aplication level + database level), besides the old objects in V7.

Personally I think we need an archive function to save old stock data in another table, redefine the stock level function based on the archive result (inventory line or another seperate table).

 

For the workflow, I do not know how to deal with it yet. Maybe we should get rid of workflow on each stock operations.

 

With the volume increases, it requires the system to lighten the resource of every operations . 

It would be great to have a proof that odoo / openerp can work  on that level volume.

Look forward to receiving more feedbacks from the community.

 

BR,

 Original Message 

Sender: Raphael Valyi<[hidden email]>

Recipient: lin.yu<[hidden email]>

Cc: OE Community [hidden email]<[hidden email]>

Date: Monday, Aug 25, 2014 10:43

Subject: Re: [Openerp-community] odoo with a daily 50000 orders volume

 

Hello Lin,

 

we worked indirectly on a case in the USA with 1/10 of the numbers you mentioned. And we had terrible issues. Just to be clear we were called as firemen and didn't had a chance to study things upfront on that project. Well basically, 2 things became extremely slow: workflows because tables were huge and stock moves (that was on 6.1) because millions of move to crawl to compute any stock level.

 

On v8, there is good hope that the stock scalability issue is gone. As for the workflows I'm afraid I'll probably still have issues: all workflows use the same tables and open workflow instances of sale, stock, account accumulate. My advice is: if it's on a version prior to v8, forget it because of the way the stock moves work. If it's on v8, well in any case plan some very serious optimizations and testing to see feasibility.

 

Number of concurrent users isn't too hard to tackle by adding more cores now that we have process based parallelism, but as for very large tables, well you'll have to clean them up often, you'll have tweak indexes, probably to bypass some computations and use some custom SQL, add caching...

I never heard about sharding in OpenERP/Odoo so I guess nobody ever did sharding, so don't assume it would be easy like in frameworks where it's being done...

 

As for the company we worked for, well, after 18 months they ended up dropping OpenERP for SAP. Probably putting way more money on it than trying again on v8 and paying a lot of engineering to get the things optimized, but still it's the decision they took and scalability was one of the main reasons and that was 1/10th of the figures you mentioned. So don't take that challenge lightly and good luck!

 

Regards.

 

 

-- 

Raphaël Valyi

Founder and consultant

<a href="tel:%2B55%2021%203942-2434" target="_blank">+55 21 3942-2434

 

https://ci3.googleusercontent.com/proxy/OLH4-qLyG0Jj61va9WwKhASRxN2uYq-XEF_FY9D2dco8CfAl9ktMPnC_SX2zIIAwVb7b8b0hXgcvTk4YLsWTY5VSNcpXfQ=s0-d-e1-ft#http://akretion.s3.amazonaws.com/assets/sign.png

 

 

On Sun, Aug 24, 2014 at 10:21 PM, lin.yu <[hidden email]> wrote:

Hello Community,

 

We have a potential lead which has a volume of daily 20000 sale orders and 50000 sale order lines from Magento, which will probably result in a daily 100000 or 200000 stock moves, 3000000 stock moves per month.

I would like to ask that has anyone experience on Odoo/OpenERP with that daily volume? How about the performance after one month?

 

I would like also to ask if you have any module on 

* archive stock move , sales order, and other objects

* sales forecasting

 

BR,


_______________________________________________
Mailing list: https://launchpad.net/~openerp-community
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-community
More help   : https://help.launchpad.net/ListHelp

 


_______________________________________________
Mailing list: https://launchpad.net/~openerp-community
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-community
More help   : https://help.launchpad.net/ListHelp


_______________________________________________
Mailing list: https://launchpad.net/~openerp-community
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-community
More help   : https://help.launchpad.net/ListHelp