How to Get the List of all Files with Size,Modified and Path from GCS Bucket and Load into BigQuery

Topic: How to Get the List of all Files with Size, Modified and Path from GCS Bucket and Load into BigQuery.

In this post, you will learn how to get the list of all files with their size, modified date, and path from a Google Cloud Storage (GCS) bucket and load them into BigQuery.

We will guide you through the process step by step, starting with setting up the necessary permissions and credentials for accessing GCS and BigQuery.

By the end of this video, you will clearly understand how to extract file metadata from a GCS bucket, load it into BigQuery, and leverage its powerful querying capabilities for further analysis.

If you’re a data engineer, data analyst, or anyone working with large datasets in Google Cloud Platform, this post is for you! 

Script: 

function listFolderContents() { var foldername = 'Final Logos'; // provide the name of Folder from which you want to get the list of files var ListOfFiles = 'ListOfFiles_' + foldername; var folders = DriveApp.getFoldersByName(foldername) var folder = folders.next(); var contents = folder.getFiles(); var ss = SpreadsheetApp.create(ListOfFiles); var sheet = ss.getActiveSheet(); sheet.appendRow( ['name', 'link','sizeInMB'] ); var var_file; var var_name; var var_link; var var_size; while(contents.hasNext()) { var_file = contents.next(); var_name = var_file.getName(); var_link = var_file.getUrl(); var_size=var_file.getSize()/1024.0/1024.0; sheet.appendRow( [var_name, var_link,var_size] ); }
};

That’s it! You should now have a BigQuery table with all the files from your GCS bucket along with their size, modified date, and path.

 

Video Demo: How to Get the List of all Files with Size, Modified and Path from GCS Bucket and Load into BigQuery.

10 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. Das Schreiben einer Masterarbeit ist eine äußerst anspruchsvolle Aufgabe, insbesondere wenn man die vielfältigen Anforderungen und die umfassende Seitenanzahl berücksichtigt. In solchen Fällen kann es vorteilhaft sein, nach Fachleuten zu suchen, die bei der Erstellung der Masterarbeit helfen können. Eine häufig genutzte Option ist die Beauftragung eines Ghostwriters- meinghostwriter.de/preise/ , der die Arbeit gemäß den individuellen Anforderungen und Wünschen verfasst. Dieser Ansatz ermöglicht es, die Herausforderungen zu bewältigen und ein zufriedenstellendes Ergebnis zu erzielen.

    ReplyDelete
  3. Ich verstehe Ihre Anmerkungen zu den Herausforderungen beim Schreiben einer Masterarbeit, insbesondere angesichts der vielfältigen Anforderungen und des umfangreichen Umfangs. In solchen Fällen kann es vorteilhaft sein, nach Fachleuten zu suchen, die bei der Erstellung der Masterarbeit helfen können. Eine häufig genutzte Option ist die Beauftragung eines Ghostwriters, der die Arbeit gemäß den individuellen Anforderungen und Wünschen verfasst. Dieser Ansatz ermöglicht es, die Herausforderungen zu bewältigen und ein zufriedenstellendes Ergebnis zu erzielen.

    ReplyDelete
  4. FASTEST WAY TO RECOVER YOUR STOLEN BITCOIN/USDT 2024

    I cannot thank ethicsrefinance enough for their incredible work in helping me recover my stolen $325,000 USDT. I was devastated when I realized my funds had been stolen, but ethicsrefinance stepped in and used their expertise to retrieve my money. Their professionalism and dedication to their work was evident throughout the entire process. They kept me updated and informed every step of the way, and I felt confident and reassured knowing that my case was in their hands. Thanks to ethicsrefinance, I was able to recover my funds and regain my financial stability. I highly recommend their services to anyone in need of ethical and reliable hacking assistance. Thank you, ethicsrefinance, for your exceptional work and for giving me peace of mind. Reach out to EthicsRefinance hackers today and reclaim what is rightfully yours

    EMAIL VIA: ethicsrefinance @ gmail .com

    TELEGRAM: @ethicsrefinance

    ReplyDelete
  5. This reseach was very helpful to me when I was setting files into BigQuery from a GCS bucket and building a list of files along with size, last modify date and guide. It helped to define what it was all about; this is the same way I searched for a dependable presentation writing service uk to ease my work. I, personally love the straightforward and well articulated instructions for following the guidelines stepwise!

    ReplyDelete
  6. Sehr hilfreicher Beitrag! Besonders die schrittweise Anleitung zur Übertragung von Dateien aus einem GCS-Bucket nach BigQuery finde ich sehr gut erklärt. Genauso transparent und klar sollten auch Ghostwriter Kosten dargestellt werden, um sicherzustellen, dass es keine versteckten Gebühren gibt und der Kunde von Anfang an weiß, was ihn erwartet.

    ReplyDelete
  7. Great tutorial! The step-by-step explanation is clear and very helpful. This is exactly what I was looking for to simplify file management tasks. Thanks for sharing! Hacopy

    ReplyDelete
  8. HARNESS THE EFFECIENCY OF CRYPTO RECOVERY-SYNACK TECH
    I came across SYNACK TECH which is an exceptional and highly reputable private licensed investigation firm based in the states that i had the privilege of engaging with after my company suffered a loss of $950,000 worth of btc due to a mistaken transfer to an unknown account by our financial accountant. Despite our desperate attempts to recover the funds through traditional channels and various recovery firms, we were met with disappointment and further financial losses. However, our fortunes changed when we discovered SYNACK TECH. Upon contacting them, their team of skilled professionals' swiftly and effectively managed to recover $830,300 of our lost funds, bringing immense relief and peace to our company. Their dedication and expertise were truly impressive, and their ability to successfully recover such a significant portion of the lost funds is a testament to their exceptional capabilities in the realm of cryptocurrency recovery. I wholeheartedly recommend SYNACK TECH for any hacking services and crypto recovery needs as their outstanding performance has not only mitigated the financial impact on our company but also restored faith in the possibility of recovering lost assets in the complex world of cryptocurrency. This experience has highlighted the invaluable importance of seeking out a reputable and skilled professionals like SYNACK TECH when faced with similar challenges, as their proficiency and commitment to their clients' success are truly unique and unmatched. With their extensive knowledge and unwavering dedication, SYNACK TECH has proven to be an invaluable asset in navigating the complexities of cryptocurrency recovery, providing hope and a lifeline for those who have experienced significant financial losses in the digital asset space. Their swift and effective actions have not only mitigated the devastating impact of our financial loss but have also restored our confidence in the possibility of recovering lost assets in the intricate world of cryptocurrency. I cannot overstate the level of gratitude and admiration we hold for SYNACK TECH, as they have not only demonstrated unmatched expertise but have also exhibited a deep commitment to their clients’ success. If you in a similar predicament reach out to SYNACKHACK {@} TECH {-} CENTER {DOT} COM

    ReplyDelete
  9. Great guide on how to get the list of all files with size, modified date, and path from a GCS bucket and load them into BigQuery. This process can really help streamline data management. By the way, for those interested in secure and seamless trading, check out Tiger Exch 247 for a top-tier trading experience!

    ReplyDelete
  10. Simplifying GCS file management while integrating with BigQuery is essential for data workflows. Curious how similar streamlined approaches benefit professionals, like LinkedIn Profile Writers Australia, in their daily tasks.

    ReplyDelete