Running this script is like mining CPU hits high 90% to 100 I have an intel Gen 10

[ufomike]

High CPU clock

When running this script my CPU is clocking at 90% plus. Is this normal or should I make any changes to get the CPU usage down?

[thefl0yd]

the program runs web browsers in the background that continually fetch web sites, scrape for information (like "in stock" or "add to cart") and make a decision based on that.

You're basically mining for a GPU (or ps5. or whatever you're looking for). So yes. high CPU usage is expected.

[jiglesiasc]

Check your power settings. My CPU is an 8th gen (8700k) and the script consumes from 20-30%, and your CPU should be stronger.

[j0um]

I also have a 8700k and I confirm the script will use all cpu resources. I'm scraping 10+ stores and all gpus.

[jiglesiasc]

I was scanning for only the 3080 in 3 stores at the lowest frequency possible. I'm not sure if scanning for that much video cards and that much stores is good, unless you run different instances of streetmerchant otherwise you'll be checking the same link each 5 minutes or more which is not going to work

[jef]

Try LOW_BANDWIDTH=true. Also for a laugh: Mac Pro 1.5 TB RAM Upgrade vs Google Chrome.

I usually ran only a few stores at a time so this doesn't happen. I'm trying to refactor a bit of the code to help mitigate and optimize this.

[wasauce]

@jef thanks for this awesome tool.

Do you have any tips for how to for example only run 1 chrome instance at a time rather than having multiple running at the same time?

Is the answer to only run against 1 store.

[KevinNovak]

I'm curious to know this too, can we specify the number of chrome instances?

[jef]

Chrome instances are also tabs. So if you only want one Chrome instance, I would only run with one store.

[jef]

This could be implemented to force this, but I don't think I'll be doing that work anytime soon.

[sgpascoe]

@jef Just wondering this myself. Is it possible to limit usage or change affinity for all instances, or slow down the scrape speed to give my CPU a rest? I'm on a 9900k and the radiator fans are going wild cooling it while it searches. I'd love to have a 'scrape limited by CPU temperature' setting, ha!

Thanks so much for the software, you're helping a great many people.

[Mr-Tech-13]

PAGE_BACKOFF_MIN=
PAGE_BACKOFF_MAX=
PAGE_SLEEP_MIN=
PAGE_SLEEP_MAX=

[marcioa6]

You can run that as a docker and limit resources, that's what I had to do to avoid 90% CPU.
Something like:
docker run -d -it --memory="2g" --memory-swap="2g" --cpus="2.0" --rm --env-file ./dotenv ghcr.io/jef/streetmerchant:nightly

It didn't run very smooth on 1G/1CPU, I was having a lot of timeouts and had to set it to over 2 minutes to avoid it, 2G/2CPU is good to start.

[jef]

Good idea!