1001 Freelance Projects
Latest Projects from
Freelance Marketplaces
View Project
View this project in detail
(Note: you will be redirected to external marketplace)
Project title:
Job Data Collection System Python (scraping)
Posted by:
External project from PeoplePerHour
Started:
18-Nov-2024 04:24 GMT
Description:
Expected duration: less than 1 week
Project Overview
We are seeking an experienced Python developer to optimize and enhance our job data collection system. The current Selenium-based approach needs to be replaced with a more efficient API-driven solution, incorporating sophisticated data management and robust error handling.

Key Requirements
- Strong Python programming skills with API integration experience
- Database design and implementation (PostgreSQL preferred)
- Experience with data versioning and delta tracking
- Familiarity with VPN handling for IP rotation
- Linux server deployment experience (Ubuntu)

Technical Specifications

Core Functionalities
1. API Integration
- Implement API-based job ID collection to replace current Selenium approach
- Design intelligent filtering system to manage data retrieval within API limitations
- Develop dynamic filter adjustment for optimal data collection

2. Database Design & Implementation
- Design and implement a PostgreSQL database structure
- Key data points to track:
- Job IDs and metadata
- First addition and update dates
- Full job details (JSON format)
- Update tracking and versioning
- Job availability status

3. Data Management
- Implement delta versioning for historical tracking
- Design system to handle regular job listing updates
- Ensure no data loss during updates

4. System Features
- Flexible time period selection for data retrieval
- Automatic filter optimization to work within API limitations
- IP rotation mechanism using NordVPN

Additional Requirements
- Comprehensive logging system
- Email notification system for errors and results
- Daily statistics tracking and reporting
- Server deployment on Ubuntu VPS

Technical Considerations
- System must handle large volumes of data efficiently
- Solution should be scalable and maintainable
- Must work within API rate limits and restrictions

Deliverables
1. Complete Python codebase
2. Database schema and implementation
3. Import of existing data
4. Deployment documentation
5. System documentation including error handling procedures

Skills Required
- Advanced Python programming
- API integration expertise
- Database design and optimization
- Linux server administration
- Network handling (VPN integration)

This is a complex project requiring a developer with strong system design skills and attention to detail. The ideal candidate will have experience with large-scale data collection and management systems.
Project ID:
3409082
Project category:
Project budget:
View this project in detail
(Note: you will be redirected to external marketplace)
Last Projects / Browse Projects
  Project Started
Need developer with previous experience in submitting app to review -- 2
Category: Android, Graphic Design, Mobile App Development, User Interface / IA, Web Design
Budget: $10 - $30 USD
26 Apr 2025 04:04 GMT
Instagram Outreach for Nail Artists
Category: Copywriting, Social Media Management, Social Media Marketing, Virtual Assistant
Budget: $30 - $250 USD
26 Apr 2025 04:04 GMT
High-Converting Fine Jewelry Shopify Store
Category: ECommerce, Internet Marketing, SEO, Shopify, Web Design
Budget: $30 - $250 USD
26 Apr 2025 04:03 GMT
On-Location Fashion Shoot in Lucknow
Category: Graphic Design, Illustration, Photo Editing, Photography, Photoshop
Budget: ₹1500 - ₹12500 INR
26 Apr 2025 03:59 GMT
Server Cleanup Specialist Needed
Category: HTML, MySQL, PHP, Server, Web Hosting
Budget: ₹1500 - ₹12500 INR
26 Apr 2025 03:57 GMT
Global Partnerships Specialist
Category: Business Analysis, Business Plans, Financial Research, Market Research, Research
Budget: $30 - $250 USD
26 Apr 2025 03:56 GMT
Modern Logo & Web Design
Category: Business Card Design, Graphic Design, Logo Design, Photoshop, Web Design
Budget: $250 - $750 AUD
26 Apr 2025 03:55 GMT
Comprehensive Server and API Setup
Category: Amazon Web Services, Linux, PHP, System Admin, Web Hosting
Budget: $30 - $250 AUD
26 Apr 2025 03:55 GMT
Wordpress. BAck up website
Category: CSS, HTML, PHP, Web Design, WordPress
Budget: $10 - $30 USD
26 Apr 2025 03:54 GMT
Nature Bites Logo Design
Category: 3D Design, Graphic Design, Illustration, Logo Design, Photoshop
Budget: ₹12500 - ₹37500 INR
26 Apr 2025 03:53 GMT
Technical Documentation Specialist Needed
Category: Amazon Web Services, Apache, Internet Security, Linux, System Admin
Budget: $250 - $750 AUD
26 Apr 2025 03:52 GMT
Building an AI agent to replace IVR system
Category: Artificial Intelligence, Low Code, VoIP, Workflow Consulting
Budget: ₹1500 - ₹12500 INR
26 Apr 2025 03:52 GMT
Android Casino App Development
Category: Android, Game Design, Game Development, IPhone, Mobile App Development
Budget: ₹75000 - ₹150000 INR
26 Apr 2025 03:49 GMT
Trading Cards E-Commerce Platform
Category: ECommerce, Graphic Design, HTML, PHP, Web Design
Budget: $30 - $250 USD
26 Apr 2025 03:49 GMT
Canva video editing. Showreel type. Files provided. Start NOW!!!!!
Category: Data Entry, Video Editing, Video Production
Budget: ₹100 - ₹400 INR
26 Apr 2025 03:49 GMT
Browse All Projects
Projects by Skills ...
android
ajax
asp
aspnet
cms
cpp
csharp
css
delphi
design
drupal
excel
facebook
flash
html
java
javascript
joomla
iphone
mysql
photoshop
php
python
ruby
seo
sql
sysadm
translate
typing
twitter
vbnet
xml
wordpress
writing
New!
Проекты на русском
(Projects in Russian)

Copyright © 2005-2024
1001 Freelance Projects