13:22 <stellar-slack> <armed10> When i send 30 000 transaction to a standalone core. they are all status: pending. but after 10 minutes there are only 26 200 records in the txhistory table of the stellar database. What could be the reason of that?

13:23 <stellar-slack> <sacarlson> are you feeding back the status as you perform the operations?

13:24 <stellar-slack> <sacarlson> I don't think you can be performing two transactions on one account if the last in in process

13:25 <stellar-slack> <sacarlson> just my guess

13:25 <stellar-slack> <armed10> The transactions are submitted to the core directly, and it returns the status.

13:25 <stellar-slack> <armed10> That could be it, I'll try doing a bunch of transactions from all seperate accounts

13:26 <stellar-slack> <sacarlson> ok and you program looks at the status results before it performs the next tx on that account

13:26 <stellar-slack> <sacarlson> ok

13:26 <stellar-slack> <sacarlson> just have it monitor before continuation on each account

13:27 <stellar-slack> <armed10> No it doesn't wait for it, but 10000 accounts are used, so it would have to submit 10000 payments before doing one from the same account

13:27 <stellar-slack> <sacarlson> ok if that's the case what was the results of the transactions that failed?

13:28 <stellar-slack> <armed10> The records don't show up in the database

13:28 <stellar-slack> <armed10> so only the status pending is returned

13:28 <stellar-slack> <sacarlson> yes no records in the database but what was the return results from the attempted tx?

13:29 <stellar-slack> <sacarlson> or you don't look?

13:29 <stellar-slack> <armed10> Just the status PENDING

13:30 <stellar-slack> <sacarlson> well after pending it should return something I would think

13:30 <stellar-slack> <sacarlson> I don't recall not getting anything but I don't do 10k tx

13:31 <stellar-slack> <sacarlson> is pending all you ever get?

13:31 <stellar-slack> <armed10> yes I determine success based on a query to the database

13:32 <stellar-slack> <armed10> It does not employ Horizon, so you have to query the results yourself

13:32 <stellar-slack> <sacarlson> oh IC well I somehow get more results than that even in the mss-server I get the reason for failure

13:33 <stellar-slack> <sacarlson> I would have to look at how I do it with mss-server that also bypasses horizon and is direct to stellar-core

13:38 <stellar-slack> <dchrist22> @ armed10 I have the same kind of behaviour. When I send ~10 tx per second over one connection, I've also get PENDING responses, but not all of them appear in the database after waiting for a few ledger closes.

13:39 <stellar-slack> <sacarlson> with a quick look I see after the tx I look the the tx in the database every 1 sec. if I fail to find it in 15 sec I consider it a time out failure

13:40 <stellar-slack> <sacarlson> otherwise I get he tx return results and decode them and return that

13:41 <stellar-slack> <sacarlson> I don't recall seeing many timeout failures but I guess it's posible. maybe wait 30 sec if no result perform the tx again

13:42 <stellar-slack> <armed10> When we only do 1 transaction per account, they all resolved fine

13:42 <stellar-slack> <armed10> So it was probably a bad sequence number

13:42 <stellar-slack> <armed10> in other words, you were right :)

13:44 <stellar-slack> <dchrist22> but with bad sequence numbers you don't get a "PENDING" response, they got rejected in the first place

13:44 <stellar-slack> <sacarlson> how do you setup for sequence numbers? you don't fetch them from database before tx?

13:44 <stellar-slack> <armed10> We do fetch them, but with 10000 transactions in between, it might not work

13:45 <stellar-slack> <armed10> The first transaction might not have been processed after the second one is submitted

13:45 <stellar-slack> <sacarlson> well it sounds like plenty of time if 10k tx before the next lookup of sequence

13:46 <stellar-slack> <sacarlson> I guess that's like 10 sec so that's not quite enuf time I leave 15 sec for timeout

13:46 <stellar-slack> <sacarlson> since I guess you can perform about 1000 tx per sec

13:47 <stellar-slack> <armed10> So for each account we would ask for a sequence number (10k accounts) then just add one for the second and one more for the third transaction of each account

13:47 <stellar-slack> <sacarlson> so setup 20k accounts that should take 20 sec maybe it will work with out any timing problems

13:48 <stellar-slack> <sacarlson> well if you note after 20 sec that the sequence number didn't change then you know something went wrong on that tx and you can perform it again

13:49 <stellar-slack> <armed10> Maybe we're just pushing the system a bit hard, these aren't exactly heavy duty servers we're testing on

13:50 <stellar-slack> <armed10> What kind of ledger size do you guys use? default or bigger?

13:50 <stellar-slack> <sacarlson> well sounds like you just get out of sync with seq I guess

13:50 <stellar-slack> <sacarlson> what's a ledger size? is that some config on stellar-core?

13:51 <stellar-slack> <armed10> yes, default it does about 400 transaction per ledger (1 ledger every 4-5 seconds)

13:51 <stellar-slack> <sacarlson> I've never played with it. just the defaults

13:53 <stellar-slack> <sacarlson> I would at least try feedback of the sequence number on each tx instead of just incrementing. then you might also detect when it starts failing

13:58 <stellar-slack> <sacarlson> oh as I read my code I'm guessing if your hitting this core hard it will probly fall out of sync with the other cores. when they are not in sync I guess the tx might take a long time to succeed if ever

14:00 <stellar-slack> <sacarlson> so on timeout you could also add a sub that checks the stellar-core status state of sync

14:02 <stellar-slack> <sacarlson> I'm not sure what the transaction return from a stellar-core out of sync is

14:17 <stellar-slack> <armed10> Well I'm testing against a standalone node currently to rule the out-of-sync possibility out

14:17 <stellar-slack> <dchrist22> are you sending over one or multiple connections?

14:17 <stellar-slack> <sacarlson> oh ok so I guess out of sync might not be posible then

14:38 <stellar-slack> <armed10> @dchrist22: I use a simple nodeJS http get, not sure what connections it uses.

14:42 <stellar-slack> <dchrist22> @armed10: okay because I am currently trying out sending over multiple connections. with a single connection I only can send ~10-15 transactions / second to the core

14:44 <stellar-slack> <sacarlson> I would assume you would would have to have multi processes running with some waiting and other setting up new transactions

14:44 <stellar-slack> <armed10> we got up to 58 last round, now we're doing a batch (20 operations per transaction)

14:45 <stellar-slack> <armed10> We only use multiple threads to generate the keypairs for the accoutns

14:45 <stellar-slack> <sacarlson> oh envelopes with 20 transactions?

14:45 <stellar-slack> <armed10> yes

14:45 <stellar-slack> <sacarlson> why not just fill it with 100 or 99 then

14:46 <stellar-slack> <dchrist22> I use for each connection one thread, but basicly without waiting

14:46 <stellar-slack> <armed10> because you can only use 20 signatures per envelope and each account needs to sign

14:46 <stellar-slack> <armed10> 1 transaction per accounts

14:46 <stellar-slack> <sacarlson> IC

14:47 <stellar-slack> <sacarlson> but you could do 5 transactions per account then with 100

14:48 <stellar-slack> <armed10> indeed

14:48 <stellar-slack> <dchrist22> thats also something what I tested. one tx with 100 operations in it, all from one account

14:53 <stellar-slack> <armed10> @dc

14:53 <stellar-slack> <armed10> Also, it wouldn't exactly mimic real-world behavior

14:59 <stellar-slack> <dchrist22> @armed10: yes, that's true. It was more a way to get many transactions into the network than mimic a real scenario

20:17 <stellar-slack> <fredolafritte> @armed10: what is the value of the configuration parameter `DESIRED_MAX_TX_PER_LEDGER` ?

20:18 <stellar-slack> <fredolafritte> also check or set `HTTP_MAX_CLIENT`

20:21 <stellar-slack> <fredolafritte> with a standalone config and the client taking more than 70% CPU on a 4-year-old mac, the number of transaction per second should be at least 350 TPS

20:28 <stellar-slack> <fredolafritte> to find out when the transaction has been included in the ledger, you could the library [pg-live-table](https://github.com/FredericHeem/pg-live-table) which monitors the postgresql table `txhistory`, no more polling, it's event based