Update to include streaming support for larger JSON files

[mm-gmbd]

The README suggests using firebase-streaming-import for files larger than 250MB. My database that I'd like to import is well over that threshold (6GB+), and with the speed of ~30seconds/MB, that ends up being somewhere around 50+ hours (and has to be run twice). Also, firebase-streaming-import hasn't been updated in almost 3 years.

I'm sure this could be improved, and should be included as a part of this package using node streams (I'd be surprised if I was the only 250MB+ user...).

[rosslavery]

This is sorely needed. There needs to be an officially-sanctioned backup restoration solution that's maintained. Our database just went past the 250mb threshold, and the ~30 seconds/mb speed of the streaming import tool is extremely painful. I can't imagine the pain of the OP dealing with a 6GB db size.

@mikelehen Is there any action happening inside Firebase to update these tools?

Edit: It appers the current master branch has streaming support implemented. Just needs a version published to npm and an update to the README.

[mikelehen]

Yeah, sorry neither of these tools is great for handling large amounts of data. If you need a full backup restore or similar, your best bet may be to reach out to Firebase Support.

That said, PRs to this project to improve its performance / usefulness are welcome.

[rosslavery]

Even just publishing the current master as an npm release would be helpful.

…

On Fri, Mar 9, 2018, 1:55 PM Michael Lehenbauer, ***@***.***> wrote: Yeah, sorry neither of these tools is great for handling large amounts of data. If you need a full backup restore or similar, your best bet may be to reach out to Firebase Support. That said, PRs to this project to improve its performance / usefulness are welcome. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#29 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAoF9FZVPCTXjFvnGdaSncuZ6OZXqQy2ks5tctAmgaJpZM4PHVmu> .

[mikelehen]

Done. Sorry for the delay on that!

[philshem]

@mm-gmbd the python firebase-streaming-import package that the readme here points to seems actually to be abandoned, including memory leaks. I'll open an issue to ask to remove it from the docs here.

I've solved my problem by splitting one BIG json into multiple smaller files, and then looping over each smaller one with bash. The challenge then is to design a json structure and import process that doesn't overwrite itself after each import step.

See my question/answer on stackoverflow, related to how to load multiple files without overwriting the previous

[yeldarby]

@mikelehen - support is still directing people to the broken firebase-streaming-import package.

What's the point of having backups if you can't restore from them?

[siege-nnn]

Happy to look at introducing parallel / multi thread approach to tackle this. How many users this solves a problem for? (doesn't look like many face the issue or at least report this)

[yeldarby]

Parallelism doesn’t really help. You hit artificial rate limits pretty quickly.

[siege-nnn]

If its a problem, we can try to find a proper/working solution. Whats the alternative?

[yeldarby]

@siege-nnn Firebase support followed up with me after I posted in their Google Group:

We know that importing over 250MB is difficult and our recommended way is indeed to do it via Support Channel. Internally we use our system backups that bypasses the front-end and doesn't have all the limits imposed to our customers.

I'll make sure that all our support playbooks are updated.

We take daily snapshots of all customer's Firebases through our backend systems, and we can restore your Database to one of these snapshots. You just need to provide a date/time, and we will restore your database's data to the most recent snapshot before this given date/time. I'll need you to fill in the following information, which will serve as written verification of the restore. This must be sent from the email address that owns the firebase instance.

Please note that the database needs to be wiped before we can do the restore. During the restore, your database will be paused and unavailable.

And they attached a form I needed to fill out for them to do the restore.