VultureV1

Drawfriend pics scrapper (working but wonky )

Recommended Posts

I downloaded about a gig of ponies from past drawfriend threads with a script I wrote . Here it is - https://github.com/Vriska/PonyScraper- . It is not very userfriendly and has a few bugs that need to be ironed out but well .. it works (kinda slowly though ) . I'm just a newbie so feel free to laugh at the ducktaped code but I'd be grateful if some experienced coder can help me out . 

If you just wanted to try the script , ignore the code and download the exe . Read the readme on the page though , this thing is not userfriendly at all .

Also , the server load on EQD will be minimal as this downloads from deviantart which will end up taking most of the load .  

Share this post


Link to post
Share on other sites
3 hours ago, VultureV1 said:

I downloaded about a gig of ponies from past drawfriend threads with a script I wrote . Here it is - https://github.com/Vriska/PonyScraper- . It is not very userfriendly and has a few bugs that need to be ironed out but well .. it works (kinda slowly though ) . I'm just a newbie so feel free to laugh at the ducktaped code but I'd be grateful if some experienced coder can help me out . 

If you just wanted to try the script , ignore the code and download the exe . Read the readme on the page though , this thing is not userfriendly at all .

Also , the server load on EQD will be minimal as this downloads from deviantart which will end up taking most of the load .  

 

I once created something similar for Rainmeter. I always wondered if CloudFlare or any other DDoS protection would go haywire on your IP after sending thousands of requests within a short timeframe to download pictures.

Any reason why you created that in Python?

Share this post


Link to post
Share on other sites
29 minutes ago, Alpha268 said:

I once created something similar for Rainmeter. I always wondered if CloudFlare or any other DDoS protection would go haywire on your IP after sending thousands of requests within a short timeframe to download pictures.

Any reason why you created that in Python?

THats unlikely unless you have super fast internet because it takes time to download before requesting another pic. But in case you have that fast a connection I added some 300 ms delay. Which should be enough I think.

 

Talking about cloudflare there's this website called kisscartoon where I used to watch mlp episodes. I thought I'd automate the downloading using just basic stuff but cloudflare kept blocking me and the only output I got was "bot detected" thing. It's pretty easy to bypass, there's this module called csflare (or ckflare) that makes it go away forever. I have the program in my profile if you're interested.

 

I Used python because it's the only language I'm good at and also because of the beautifulsoup library, which is perfect for this task. Python is actually the most commonly used scraping language from what I've seen. Which language did you use for the rainmeter thing?

 

PS: did the program work?

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now