Kdenlive   bug tracker Home page

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0001536KdenliveRenderingpublic2010-03-28 20:452010-03-31 16:06
ReporterBlessJah 
Assigned To 
PrioritynormalSeverityfeatureReproducibilityhave not tried
StatusnewResolutionopen 
Platform32 bit intel and alikeOSKubuntu LinuxOS Version9.10
Product Version0.7.5 
Target VersionFixed in Version 
Summary0001536: Rendering autosave and recovery
DescriptionI'm thinking about rendering auto save.

I've got huge project, I'm rendering 10 hours, I have 99% and... CRASH DETECTED. I have to start from 0%, recovering partly rendered file afaik is impossible now.

What about rendering small parts (5 minutes) and saving them in temporary directory? After crash I could easily recover my 10 hours of rendering, only last part will be broken. After rendering whole film parts will be connected in ready file. I could also stop rendering in every moment, loosing only last small part, shut down computer and start rendering again next day.

Second way allowing to recover rendering after crash or stop, is rendering 5 minutes part and saving it to ready file. We render, but don't save rendered video to file immediately. We save add to end of file small parts, when rendering stops/crashes, we lost only last rendered part saved in RAM, and we know where we should start rendering after recover.

1st way is better, because it can crash when we're saving rendered video to file, when we have every part in separate file we can easily recover everything rendered before crash.
TagsNo tags attached.
Build/Install Method(select)
Attached Files

- Relationships

-  Notes
(0004887)
xzhayon (developer)
2010-03-30 14:08

i've discussed this issue with submitter on irc
i think that, if this can work, the first 5-minutes renderings have to be done without any kind of compression (well, mlt is not able to do lossless rendering... let's say, then, with an i-frame only profile)
i don't know if different videos can be joined without any loss, but in the past i've joined some with `cat` (followed by `ffmpeg -vcodec copy` to fix seeking) and the result was good

people (and ddennedy!): what do you think?
(0004892)
BlessJah (reporter)
2010-03-30 16:53

Afaik now movie is periodically written to an uncomplete file. Why won't we write it to tmp binary files, that cannot be played in any movie player, but after rendering can be easily connected into ready movie?

This shouldn't be hard, just save parts (for example every 100MiB) to different files instead of one uncomplete file.

This option doesn't need any combination with compression, i-frame, anything.
(0004894)
xzhayon (developer)
2010-03-30 17:02

> This option doesn't need any combination with compression, i-frame, anything.

i'm not sure. if you restart rendering a broken file, you'll have to start from an i-frame. if the whole video did not suppose to have an i-frame there, perhaps it could complain. but, again, ddennedy or someone informed should reply to this

ah, for your information: i see this as an advanced option, disabled by default
(0004895)
BlessJah (reporter)
2010-03-30 17:31

Ofc disabled by default. But maybe enabled (by default) when rendering gonna take more than 1 hour? (2 hours? 5?)

>i'm not sure. if you restart rendering a broken file, you'll have to start from an i-frame.

I'm not developer, so i can be wrong.
(0004897)
thatonefilmguy (reporter)
2010-03-31 04:41

I made a comment on this before but my phone decided not to actually commit it.

I'm not a developer, but this sounds like an incredibly unnecessary 'feature' which will just end up introducing more bugs into the rendering process. The proper way to resolve this issue is for the rendering to be stable and not crash :o)

There are numerous problems with this IMO.

1. This would make the rendering process incredibly more complex, because not only would the render need to complete successfully, it also means that it would then have to successfully stitch the video together after rendering completes. This would introduce a vast number of new places for defects to arise which could crash/corrupt the export/stitching process and just make it more unstable.

2. Stitching together video files will differ from format to format. Different formats may or may not be 'stitchable' (assuming it is feasible at all). Each format would probably have to be handled in a different manner, which would make the process incredibly complex.

3. The disk space required to export a video would double. Lets suppose you render a 2 hour DV video file. DV takes up approximately 13GB per hour. To render a 2 hour file, you normally need 26GB of disk space. To render the video to a set of temporary files, this means you will have 26GB of temp files + 26GB for the 'stitched' file. If you write temp files, stitch them, then delete the temp file you can use less space, but then you risk losing data if the stitching process fails for one of the pieces then you no longer have the backup/temp files and you lose the stitched file as well (thus defeating the purpose).

4. Rendering/export time would be greatly increased, since it would have to render the file once, and then stitch all the pieces together after rendering completes.

5. Someone above mentioned that the renderings have to be done without any compression. A 5-minute uncompressed file would be enormous. Even for standard definition video, the files would be large. A 1920x1080 HD video frame is approximately 6 times the size of standard definition, thus would be approximately 6 times as large of a file for uncompressed video. Also, this means either the video then has to be recompressed during the 'stitching' process, or the final output must also be uncompressed video. More places to create and find more bugs.

There are other issues besides the above, but those are just a few. Bottom line, I think this would be a very risky attempt to insure partial exports can be recovered, when the real issue is the crash itself.

A better way to render in segments is to set an IN/OUT on the sequence in kdenlive, then render one section at a time until the entire sequence has been exported. Then stitch the files together yourself afterwards either in another kdenlive project or another method such as ffmpeg. This way if you have an effects heavy sequence that is taking a long time to export, you can export it in pieces and then exporting the second 'stitched' sequence containing the individual zones will take much less time since the effects do not need to be rendered again.
(0004898)
xzhayon (developer)
2010-03-31 08:30

> I'm not a developer, but this sounds like an incredibly unnecessary 'feature'
> which will just end up introducing more bugs into the rendering process. The
> proper way to resolve this issue is for the rendering to be stable and not
> crash :o)

but a crash in kdenlive is not the only possible problem. actually, it's not the most probably too
 
> 2. Stitching together video files will differ from format to format.
> Different formats may or may not be 'stitchable' (assuming it is feasible at
> all). Each format would probably have to be handled in a different manner,
> which would make the process incredibly complex.

the idea is to render to a common "good" format, then join the files and convert them to the requested format
 
> 4. Rendering/export time would be greatly increased, since it would have to
> render the file once, and then stitch all the pieces together after rendering
> completes.

of course, that's why i thought of it as an advanced feature
 
> A better way to render in segments is to set an IN/OUT on the sequence in
> kdenlive, then render one section at a time until the entire sequence has
> been exported. Then stitch the files together yourself afterwards either in
> another kdenlive project or another method such as ffmpeg. This way if you
> have an effects heavy sequence that is taking a long time to export, you can
> export it in pieces and then exporting the second 'stitched' sequence
> containing the individual zones will take much less time since the effects do
> not need to be rendered again.

that's what i thought ;)
(0004900)
thatonefilmguy (reporter)
2010-03-31 13:43

> the idea is to render to a common "good" format, then join the files and convert them to the requested format

Yes but then you are recompressing the video frames twice, which will result in more compression artifacts than exporting a single time (if you're a professional editor then that is a big deal). The quality of the 'good' format chosen could have a huge impact on the quality of the final output format. Real 'stitching' would need to involve rewrapping the video files together from multiple video files into a single video file without recompressing them IMO. It is theoretically possible to do this - Final Cut Pro has export optimizations which do not require all video frames to be recompressed if the sequence and output format is the same as the original format of the clip and if FCP supports rendering the format this way - however, it is not at all a minor feature, and in the FCP example they are doing it all during a single export process for performance sake (decreasing export time) rather than doing it in a secondary process to make a workaround for existing bugs (increasing export time).

In that sense, I think this would be a great feature for exporting video from a sequence by rewrapping it and skipping recompression (if possible) because it could greatly expedite render time, but doing it for the sake of being able to recover a partially exported file seems like the wrong way to go about 'fixing' an existing render bug, whether it be in kdenlive, mlt ffmpeg, etc.

In any case, the bottom line IMO is that this will certainly introduce more bugs. I (personally) think it is better to fix existing bugs in order to fix crashes, rather than implement new features which will only result in more places for crashes to occur which will also inevitably need to be fixed. But then again I'm not a developer so it is not up to me!

> that's what i thought ;)

If we agree on that then I think that more or less solves the problem you are having? :oD
(0004901)
BlessJah (reporter)
2010-03-31 16:06

>Afaik now movie is periodically written to an uncomplete file. Why won't we write it to tmp binary files, that cannot be played in any movie player, but after rendering can be easily connected into ready movie?

That is second way to to that, just change method of saving uncompletely file.
This way gonna need 2x space, this adds new places that could crash, this gonna need more time, this will be harder.
But now kdenlive likes to crash, so we need autosaves, and we need recovery system.

>it is better to fix existing bugs in order to fix crashes, rather than implement new features which will only result in more places for crashes to occur

You're 100% right, but not only kdenlive can crash. Energy provider, my dog, storm, everything can cause unexpected shutdown. Developers won't fix power plant and won't teach my sister not to use computer when I'm rendering sth.

Fixing bugs is the most important, but we shouldn't think only about bugs.

- Issue History
Date Modified Username Field Change
2010-03-28 20:45 BlessJah New Issue
2010-03-30 14:08 xzhayon Note Added: 0004887
2010-03-30 14:08 xzhayon Status new => feedback
2010-03-30 16:53 BlessJah Note Added: 0004892
2010-03-30 16:53 BlessJah Status feedback => new
2010-03-30 17:02 xzhayon Note Added: 0004894
2010-03-30 17:31 BlessJah Note Added: 0004895
2010-03-31 04:41 thatonefilmguy Note Added: 0004897
2010-03-31 08:30 xzhayon Note Added: 0004898
2010-03-31 13:43 thatonefilmguy Note Added: 0004900
2010-03-31 16:06 BlessJah Note Added: 0004901


Copyright © 2000 - 2014 MantisBT Team
Powered by Mantis Bugtracker