WP Content Crawler – Get content from almost any site, automatically v1.9.0
WP Content Crawler – Get content material from nearly any web site, robotically v1.9.0
WP Content Crawler is a SEO Plugin its very user friendly indexing in search engine. Get content material from nearly any web site to your WordPress weblog, robotically!
FOR WHAT IT CAN BE USED
- Create a private web site which collects information, posts, and many others. out of your favourite websites to see them in a single place
- Use it with WooCommerce to gather merchandise from procuring websites
- Gather merchandise from affiliate packages to earn a living
- Gather posts to create a take a look at atmosphere on your plugin/theme
- Gather plugins, themes, apps, photos from different websites to create a set of them
- Maintain observe of rivals
- You possibly can think about something. The web is filled with contents
HOW IT WORKS
It’s all about CSS selectors and you’ll learn to use them in minutes by watching the introduction tutorial. The plugin’s visible inspector device additionally helps you discover CSS selectors simply by clicking onto the weather within the goal websites.
|Save each submit element
Title, excerpt, content material, tags, classes, slug, date, customized meta, taxonomies, meta key phrases, meta description, featured picture, submit photos, standing… Simply all the things.
|Visible selector (visible inspector)
Simply click on to a component to seek out its CSS selector. You may as well get different CSS selectors that you simply may be considering. There isn’t any want to go away your admin panel anymore.
|Crawl (scrape, seize, save) posts
After the settings are configured, the plugin finds URLs of the posts and crawls them robotically within the background.
|Recrawl (replace) posts
Recrawl posts robotically to maintain them up to date on a regular basis. You possibly can restrict what number of instances a submit might be up to date, set replace interval, and ignore outdated posts.
You wish to delete outdated crawled posts? The plugin can delete them robotically.
You possibly can set what number of instances URL assortment and submit crawling occasions ought to run every time for a web site. As an example, it can save you Three posts each minute, or run URL assortment 5 instances each 2 minutes.
The goal class doesn’t exist in your web site? No downside. The plugin can create the goal classes for you. Simply outline the CSS selectors that discover class names. They’ll even be created as subcategories.
|Save slugs (permalink)
You possibly can outline the permalink of the posts. You may get the permalink from the goal web site, enter customized textual content, and even create templates for the slugs through the use of quick codes.
Save taxonomy values by retrieving them from the goal web site or coming into manually. Saving particulars of customized submit varieties is less complicated than ever.
|Save posts into customized classes
A customized submit sort has customized classes? No downside. You possibly can outline customized class taxonomies utilized by the customized submit sort and choose these classes when defining the classes of the submit. The plugin may create customized classes for you.
|Customized submit meta
Save something as customized submit meta. You need to use a CSS selector or simply sort the worth.
|Content material templates
Put together submit content material, title, excerpt, listing merchandise and gallery merchandise templates utilizing quick codes. Furthermore, you may outline templates for values of every CSS selector utilizing the choices field.
You possibly can write different selectors to get the information even when the goal web site has submit pages designed in a different way from one another.
|Discover and exchange something
You need to use plain textual content or common expressions to seek out and exchange something. You possibly can even modify the HTML of the web page, create your individual HTML components and write selectors to make use of them. You possibly can even change picture URLs. You will have the ability.
Goal submit has multiple web page? No worries. It can save you paginated posts as nicely.
|Checklist sort posts
Some websites create posts with a listing inside. You possibly can extract the listing from the submit, create a template that ought to be utilized to every listing merchandise and even reverse the listing.
|Take away pointless components
Generally you could eliminate some components, similar to commercials, feedback, you identify it. Simply write its CSS selector and it’s eliminated.
|Mechanically insert class URLs
Goal web site has lots of of classes? Piece of cake. Simply write the CSS selector and the plugin will insert them for you.
Set submit sort. It may be a submit, a web page, a product, or every other submit sort accessible in your WordPress set up.
|Take away hyperlinks
You possibly can take away hyperlinks from the submit. Simply verify the checkbox and the hyperlinks are gone. That simple.
You possibly can set a password for the posts to indicate them solely to the customers who’ve the password.
You possibly can add notes for your self to remind you issues in regards to the web site. CSS selectors, TODO listing, something.
|Take a look at all the things on the fly
Take a look at submit crawling, URL assortment, CSS selectors, common expressions, discover and exchange choices and proxies on the fly. You may as well allow caching to carry out the exams a lot sooner and cut back the requests despatched to the goal web site.
|Take a look at all of the settings of a web site without delay
Utilizing the tester, you may take a look at all choices you configured within the web site settings to ensure all the things works as you need earlier than enabling automated crawling.
Utilizing the instruments, it can save you posts manually with their URL, recrawl posts with their ID or delete already-saved URLs.
|Customized basic settings for every web site
You possibly can present customized basic settings for every submit to override them and make them appropriate for a web site.
You possibly can immediately publish the saved posts or maintain them as draft to verify them earlier than publishing.
|Save all photos in submit content material
Saving all photos within the content material of the submit is as simple as checking a single checkbox.
|Save photos as gallery
It can save you the pictures within the goal web page as gallery and supply a template for every picture to make it appropriate for the gallery library that you simply use on frontend. You may as well save the pictures as WooCommerce gallery by simply checking one checkbox.
|Any information as quick code
Get something from goal web page as a brief code and use the quick codes within the plugin’s templates to put any information anyplace you need.
Use a proxy or proxies to get content material from the websites to which your IP doesn’t have entry.
Connect cookies, similar to session cookies, to every request. By this manner, for instance, you may crawl the goal web site as in case you are logged in.
|Crawl as many posts as you need
You possibly can set what number of instances submit crawling or URL assortment CRON occasions ought to run. By this manner, you may, e.g., save 100 posts each minute. Simply watch out and think about your server’s capability.
Set CSS selectors whose values shouldn’t be empty for class and submit pages. When an empty worth is discovered utilizing these selectors, you will get an e-mail notification.
|Get information from JSON
Whenever you allow JSON parsing for a CSS selector, you will get the values from the JSON simply.
|Superior HTML manipulations
Discover-replace in response HTML, discover and exchange in aspect attributes, alternate aspect attributes, take away aspect attributes, manipulate HTML of a component, take away HTML components…
Use the substitute intelligence of Google Cloud Translation API, Microsoft Translator Textual content API, Yandex Translate API or Amazon Translate API to robotically translate the posts. Word that these are paid providers, apart from Yandex Translate API. The paid ones additionally provide the service without spending a dime for a restricted period of time. You possibly can see their pricing pages to be taught extra.
Use spinning to robotically rewrite crawled posts’ contents to enhance search engine marketing. The plugin at present implements solely Spin Rewriter API, which is a paid service. You possibly can go to their web site to be taught the pricing particulars.
|Duplicate submit verify
Examine duplicate posts by URL, submit title and/or submit content material. In case you are utilizing WooCommerce, merchandise whose SKU already exists are thought of as duplicate and they won’t be added to your web site.
You possibly can add/take away minutes to/from the submit date. By this manner, you may schedule submit publishing.
|Save WooCommerce merchandise
Save worth, stock, delivery, attributes, and superior choices. It can save you the product as a easy or an exterior product. You may as well set downloadable file choices and outline the product as digital. The choices can be found for WooCommerce variations higher than or equal to three.3.
You will have the management! Outline many choices for the values discovered by a CSS selector. The choices embrace find-replace, calculation, template, and JSON parsing settings. You possibly can simply import/export the choices outlined within the choices containers as nicely.
|Deal with recordsdata like a professional
Rename, copy, and transfer saved recordsdata simply. You may as well outline title, description, caption, and alt texts for the saved media recordsdata utilizing templates wherein you need to use any quick code. Additionally it is potential to offer random names to the saved recordsdata.
|Deal with iframes and scripts like a professional
WordPress doesn’t enable displaying iframes and scripts since they pose a safety danger. You possibly can flip iframe and script HTML components into quick codes by simply checking a checkbox. The quick code will present iframes and scripts from the allowed supply domains outlined by you.
With fast save button, it can save you the settings far more rapidly. No want to attend for web page to reload.
Outline common expressions in find-replace choices to find-replace something. You may as well use delimiters and modifiers to match extra exactly.
|Save “srcset” attributes
When different sizes of the saved photos can be found, the plugin assigns them into srcset attribute of img components in order that your pages will load sooner in numerous display screen sizes.
|Save “alt” and “title” attributes
Whenever you save photos, their “alt” and “title” attributes are robotically retrieved from the goal web site and assigned to the saved media. You may as well outline templates for them to use your search engine optimisation methods.
Be taught when there’s a downside. The plugin will present you the main points of the error in an effort to repair it instantly.
|Deal with character encoding issues
The plugin is ready to deal with totally different character encodings, even when the goal web site comprises combined encodings. You possibly can convert the encoding by checking a single checkbox.
|Navigate between settings simply
Repair navigation to the highest! The plugin shops the place you have been earlier than switching to a brand new tab and restores your earlier location once you activate that tab once more. No extra getting misplaced among the many settings.
|Guide crawling device
With guide crawling device, save a number of posts by coming into their URLs. You may as well enter class URLs in order that the device can get submit URLs from there. Furthermore, you may set it to crawl a number of posts on the similar time.
|Add URLs to the database
The plugin collects URLs robotically. Nonetheless, in order for you it to crawl solely sure URLs, you may add them to the database manually utilizing the guide crawling device. By this manner, the desired URLs will likely be crawled utilizing your scheduling choices, robotically.
|Allow/disable automated crawling for a particular web site
You possibly can allow or disable automated crawling for every web site individually.
You possibly can import and export web site settings simply. Simply copy and paste the code created by the plugin.
Add limitless websites and activate what number of of them you need.
See what’s occurring within the background. Energetic websites, variety of posts crawled, variety of posts up to date, final crawled and up to date posts, final added URLs, final and subsequent run of CRON occasions, at present being saved posts and URLs…
|Get updates out of your admin panel
You possibly can replace the plugin with only one click on every time an replace is prepared. Simply go to your updates web page in your admin panel.
|Use probably the most safe PHP
The plugin helps the most recent variations of PHP.
|Use probably the most trendy browsers
The plugin helps Chrome, Firefox, Safari, Opera, and Edge.
You possibly can verify the net documentation everytime you really feel a necessity.
|Fast guides proper subsequent to the settings
Every setting within the plugin has a fast information that may provide help to perceive what every setting does.
Watch video tutorials to simply learn to use the plugin.
|Able to translate
You possibly can translate the plugin into your individual language utilizing Poedit.
|Necessities||PHP >= 7.2, json, mbstring, curl, dom, WP-Cron. These are already accessible in most hosts. Even when the extensions aren’t already lively, most internet hosting websites allow you to allow these from their management panel. See the documentation for extra info.|
|Examined with WP variations||5.3, 5.2, 5.1, 5.0, 4.9|
|Examined with WooCommerce variations||3.8, 3.7, 3.6, 3.5|
|Languages||English, Türkçe, Français (partial)|