How to Write Python Scripts to Analyze JSON APIs and Sort Results



In this Python Programming Tutorial, we will be learning how to grab data from a JSON API, parse out the information we want, and then sort the data using a custom key. The API we will be using is a JSON API for Homebrew Packages and we will be sorting the packages by their popularity. We cover a lot of topics in this tutorial. We will be using the Requests Library, converting to/from JSON, reading and writing to files, writing our own sorting function, and more. Let’s get started…

Homebrew Packages –
Homebrew Analytics –

Requests Tutorial –
JSON Tutorial –
Sorting Tutorial –
List Comprehensions –

✅ Support My Channel Through Patreon:

✅ Become a Channel Member:

✅ One-Time Contribution Through PayPal:

✅ Cryptocurrency Donations:
Bitcoin Wallet – 3MPH8oY2EAgbLVy7RBMinwcBntggi7qeG3
Ethereum Wallet – 0x151649418616068fB46C3598083817101d3bCD33
Litecoin Wallet – MPvEBY5fxGkmPQgocfJbxP6EmTo5UUXMot

✅ Corey’s Public Amazon Wishlist

✅ Equipment I Use and Books I Recommend:

▶️ You Can Find Me On:
My Website –
My Second Channel –
Facebook –
Twitter –
Instagram –

#Python #JSON #API

Nguồn: https://benjaminjcohen.com/

Xem thêm bài viết khác: https://benjaminjcohen.com/cong-nghe/

25 Comments

  1. We're covering a lot in this video. We're going to see how to request data from a JSON API, parse out the information we want, sort the data using a custom key, and a lot more. We're going to be using the Homebrew API, but you don't need to use Homebrew or a Mac to follow along. This can apply to many other APIs you'd like to analyze with Python.

    As I mention in the video, our final result does have a little more functionality than what we can get from their existing API, so I believe it was definitely a useful exercise. For example, we can filter popular packages by a keyword in the description, which is a nice edition. Hope you all find this helpful!

  2. How you dont have 1 million subscribers is just ridiculous

  3. Really corey your tutorials are different they make my day.💕

  4. i need some help : how to comment out multiple python lines at once in sublime text ?

  5. github repo link please?

  6. one of my all-time favorite videos.

  7. Hye Corey, do u have this code in ur github page both python scripts and json file. Cause i got some error when try to grab all the json api. Thanks

  8. when I include the break statement, it outputs the json file, but without the break, it doesn't output the json file. Why would that occur?

  9. Hi Corey! Thank you so much for your tutorial. All of them!
    I am facing an issue, tough. One of the packages contains a different name for the "install_on_request", like this one: https://formulae.brew.sh/api/formula/zrepl.json. So, I had to wait for 90 min till load all the packages to figure this out! Although you've mentioned that is a good idea to break the loop with 'break' to check for possible errors this one got undetectable. Can you help me to solve this issue, please?

  10. Thanks for the training! Just downloaded 4997 records today in 39 mins. No mods needed to your scripts.

  11. Could you please make a tutorial on how to combine/merge two (or more) sorted json files?

  12. great teaching, great work

  13. Hello Corey, I Need your help on this. I have an endpoint basically i have to hit the api to get the data, the response coming from api is in json format.

    Below is the approach i have followed.

    import requests, json

    response = requests.get(url, accept, auth=(username, password)).content

    print(type(response)) is <class 'bytes'>

    response from the api is

    {'d': {'results': [{'__metadata': {'uri': "https://my56776.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')", 'type': 'c4codata.CorporateAccount', 'etag': 'W/"datetimeoffset'2020-02-17T13%3A24%3A23.5795610Z'"'}, 'ObjectID': '00163E6CDFCB1EDA94B110D585766BA9', 'AccountID': '1015739', 'UUID': '00163E6C-DFCB-1EDA-94B1-10D585766BA9', 'ExternalID': '', 'ExternalSystem': '', 'RoleCode': 'BUP002', 'RoleCodeText': 'Prospect', 'LifeCycleStatusCode': '2', 'LifeCycleStatusCodeText': 'Active', 'DUNSID': '', 'LegalFormCode': '', 'LegalFormCodeText': '', 'CustomerABCClassificationCode': '', 'CustomerABCClassificationCodeText': '', 'NielsenRegionCode': '', 'NielsenRegionCodeText': '', 'IndustrialSectorCode': '', 'IndustrialSectorCodeText': '', 'ContactPermissionCode': '', 'ContactPermissionCodeText': '', 'BusinessPartnerFormattedName': 'Sample Nick 1', 'Name': 'Sample Nick 1', 'AdditionalName': '', 'AdditionalName2': '', 'AdditionalName3': '', 'CurrentDefaultAddressUUID': '00163E6C-DFCB-1EDA-94B1-10D58576EBA9', 'FormattedPostalAddressDescription': '1 street1 / city1 DN16 1 / GB', 'CountryCode': 'GB', 'CountryCodeText': 'United Kingdom', 'StateCode': '', 'StateCodeText': '', 'CareOfName': '', 'AddressLine1': '', 'AddressLine2': '', 'HouseNumber': '1', 'AdditionalHouseNumber': '', 'Street': 'street1', 'AddressLine4': '', 'AddressLine5': '', 'District': '', 'City': 'city1', 'StreetPostalCode': 'DN16 1', 'County': '', 'CompanyPostalCode': '', 'POBoxIndicator': False, 'POBox': '', 'POBoxPostalCode': '', 'POBoxDeviatingCountryCode': '', 'POBoxDeviatingCountryCodeText': '', 'POBoxDeviatingRegionCode': '', 'POBoxDeviatingRegionCodeText': '', 'POBoxDeviatingCity': '', 'TimeZoneCode': 'GMTUK', 'TimeZoneCodeText': '(UTC+00:00) Greenwich Mean Time with daylight saving time (UK)', 'Building': '', 'Floor': '', 'Room': '', 'Phone': '', 'Mobile': '', 'Fax': '', 'Email': '', 'WebSite': '', 'LanguageCode': 'EN', 'LanguageCodeText': 'English', 'BestReachedByCode': '', 'BestReachedByCodeText': '', 'OrderBlockingReasonCode': '', 'OrderBlockingReasonCodeText': '', 'DeliveryBlockingReasonCode': '', 'DeliveryBlockingReasonCodeText': '', 'BillingBlockingReasonCode': '', 'BillingBlockingReasonCodeText': '', 'SalesSupportBlockingIndicator': False, 'LegalCompetenceIndicator': True, 'RecommendedVisitingFrequency': '', 'VisitDuration': '', 'LastVisitingDate': None, 'NextVisitingDate': None, 'LatestRecommendedVisitingDate': None, 'OwnerID': '210', 'OwnerUUID': '00163E6C-DFCB-1EE9-AFB3-EBCD9CCDE0F5', 'CreationOn': '/Date(1581945787675)/', 'CreatedBy': 'Mathias Huber', 'ChangedOn': '/Date(1581945863579)/', 'ChangedBy': 'Mathias Huber', 'NormalisedPhone': '', 'NormalisedMobile': '', 'EntityLastChangedOn': '/Date(1581945863579)/', 'ETag': '/Date(1581945863579)/', 'CL6CustomerCreationInERPID_SDK': '', 'CL6CustomerName_SDK': '', 'CL6CustomerCreationInERPcontent_SDK': None, 'NPGLAccExtID_KUT': '', 'NPGLAmbientcontent': '0.00000000000000', 'NPGLAmbientunitCode': 'P1', 'NPGLAmbientunitCodeText': 'Percent', 'NPGLAssociations': '', 'NPGLAssociationsText': '', 'NPGLChilledcontent': '0.00000000000000', 'NPGLChilledunitCode': 'P1', 'NPGLChilledunitCodeText': 'Percent', 'NPGLCreditLimitcontent': '0.000000', 'NPGLCreditLimitcurrencyCode': '', 'NPGLFoodGrade_KUT': '', 'NPGLFoodGrade_KUTText': '', 'NPGLFrozencontent': '0.00000000000000', 'NPGLFrozenunitCode': 'P1', 'NPGLFrozenunitCodeText': 'Percent', 'NPGLIndustryCode1': '', 'NPGLIndustryCode1Text': '', 'NPGLMembership': '', 'NPGLMembershipText': '', 'NPGLNestlEmployeeNumber': '', 'NPGLNumberofoutlets': '0', 'NPGLOthercontent': '0.00000000000000', 'NPGLOtherunitCode': 'P1', 'NPGLSAPCustomerNumber_KUT': '', 'NWUSDepartment': '', 'CorporateAccountAddress': {'__deferred': {'uri': "https://my56789.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')/CorporateAccountAddress"}}, 'CorporateAccountAttachmentFolder': {'__deferred': {'uri': "https://my56789.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')/CorporateAccountAttachmentFolder"}}, 'CorporateAccountHasContactPerson': {'__deferred': {'uri': "https://my56789.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')/CorporateAccountHasContactPerson"}}, 'CorporateAccountIdentification': {'__deferred': {'uri': "https://my56789.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')/CorporateAccountIdentification"}}, 'CorporateAccountInternationalVersion': {'__deferred': {'uri': "https://my56789.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')/CorporateAccountInternationalVersion"}}, 'CorporateAccountSalesData': {'__deferred': {'uri': "https://my56789.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')/CorporateAccountSalesData"}}, 'CorporateAccountSkills': {'__deferred': {'uri': "https://my567898.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')/CorporateAccountSkills"}}, 'CorporateAccountTaxNumber': {'__deferred': {'uri': "https://my56788.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')/CorporateAccountTaxNumber"}}, 'CorporateAccountTeam': {'__deferred': {'uri': "https://my56889.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')/CorporateAccountTeam"}}, 'CorporateAccountTextCollection': {'__deferred': {'uri': "https://my56768.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')/CorporateAccountTextCollection"}}, 'CorporateAccountVisitingHours': {'__deferred': {'uri': "https://my56789.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')/CorporateAccountVisitingHours"}}, 'CorporateAccountVisitingInformationDetails': {'__deferred': {'uri': "https://my56897.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')/CorporateAccountVisitingInformationDetails"}}, 'OwnerEmployeeBasicData': {'__deferred': {'uri': "https://my568778.crm.ondemand.com/sap/c4c/odata/v1/c4codataapi/CorporateAccountCollection('00163E6CDFCB1EDA94B110D585766BA9')/OwnerEmployeeBasicData"}}}]}}

    Now i have to flatten this data to structured format automatically without adding/selecting the attributes in the code.

    I have tried json_normalize but the output is not in proper way.

    json_response = json.loads(response)
    json_normalize(jsonResponse['d'])

    Could you please help me on this.

  14. For those who are coding along with jupyter notebook, open the notebook from the command line with these flags in order to avoid errors of time and data limits:

    jupyter notebook –NotebookApp.iopub_data_rate_limit=1e20 –NotebookApp.rate_limit_window=1000

    Thank you so much for all these videos Corey.

  15. I saw all video ads. Feel nice!

  16. Great video Corey! Would love to see a video where you work with Tweepy, and its output. Maybe leveraging the ‘trends’ feature, and or pulling a user’s timeline and making sense of the output.

  17. Hi there @Corey Schafer I have a question if for example you have two different json websites containing information information about the same product like one site contains name the other price after scraping how can I put them next to each other row by row?

  18. This was great Corey, thanks a lot for that! 🙂

  19. This was one of your best videos to date. I love the longer format using a real world project that can be applied to many other tasks.
    I hope enough people are donating and supporting you to keep up this work.

  20. Amazing video on python. So much of stuff in a single video

  21. Thank you so much! Best one on this topic I found!

Leave a Reply

Your email address will not be published. Required fields are marked *