Hello, could we use this thread to make a statement of what needs to be done for a better support of AVCHD camcorders in FFmpeg, MLT and Kdenlive. Then we may ask for changes in FFmpeg and try to find collaborative support. Dan, what do you think would be needed for a better support of AVCHD camcorders in FFmpeg?
I believe that if we write a public statement and ask for help at FFmpeg, the core hackers can concentrate on FFmpeg important issues and fix them.
Jean-Michel
- Login or register to post comments
- 6168 reads
- Flag as offensive
- PDF version
The FFmpeg developers know the situation, but are not eager to resolve it on their own instead relying upon contributed patches. I am in the same boat as them. I have enough issues to deal with, and I do not own an AVCHD camera. Therefore, I am not inclined to work on this either. This is not out of malice; it is a matter of priorities and lack of personal nuisance. I have no personal "domination" agenda nor the energy to make kdenlive all things for all people in short time.
AFAIU, to give a technical summary, the FFmpeg seek API needs to be reworked to accept PTS instead of DTS. Then, MLT can change to use PTS as well. This info is based on the last outstanding patches from Ivan Schreter who was working on this. He had a partially complete FFmpeg patch. Again AFAIU, this is a difficult situation because I believe many demuxers will need to be updated. For example, the DV demuxer would crash as a result of the patch, and some other formats were not accurate IIRC. Also, there is a new seek API in progress, and this might be part of that.
The other option is to not use FFmpeg for AVCHD support. I already need to add a separate Ogg Theora module to get better support for that. Perhaps gmerlin-avdecoder is an option or gstreamer. I do not know what is good and accurate at seeking for AVCHD. If someone can research that and give some feedback, that would be nice.
FWIW - AVCHD seek is actually much improved in 0.7.4 now that I've gotten a working build.
I'm using the Ubuntu package described at
http://www.kdenlive.org/forum/kdenlive-074-launchpad-debs-jaunty-and-int...
Quoting from my remark there on that thread -
I'm using 1080p30 .MTS footage direct out of a Canon HF100. Previously it worked alright but you couldn't seek to midway into a clip and play back, you'd just get a grey screen. Now it's still dropping b-frames so a bit jerky still, but you can definitely seek and play. Very nice.
At this point there are a couple of issues for me with AVCHD
First is playback. I'm on a dual core Athlon 64, and I'm using an Nvidia 9400 card, driver with VDPAU extensions. With this hardware, it's already a pretty impressive achievement that you can playback raw AVCHD directly in the NLE at all, even with dropped b-frames, because I was unable to do this on Windows with this hardware and a commercial NLE like Premiere Elements: not a usable editing experience, way too jerky and slow. 0.7.4 is pretty darn good in this respect. A luma transition with title over is even pretty fluid, which is pretty impressive. So maybe I'm wrong, but I'm wondering just how much better editing playback can get given hardware limitations.
Probably to me the more attractive target would be pass-through rendering back to h264 or whatever is most compatible with AVCHD. It would be really fantastic not to do a re-encode on those stretches of footage that are trimmed but otherwise unaltered or that only have audio mods. Only transitions, effects, and compositing would need re-encoding. Rendering could conceivably be extremely fast, and moreover - this would be editing with basically no generational loss: extraordinary. Additionally, you could stick a finished render back on the camera and play that back on any tv through HDMI. Movies played back directly from the camera's dedicated decode hardware look better than anything I've so far experienced even over a computer with HDMI out.
Thanks for the encouraging feedback. In addition to poor support for seeking on AVCHD we also have poor support for Ogg Theora, which is growing important due to its HTML5 support in Firefox, Opera, Chrome, and other Webkit derivatives. To that end, I am looking at adding a gstreamer or gmerlin-avdecoder backend to MLT. I know gmerlin has better support for AVCHD than we do, and I know gstreamer has better support for Ogg Theora. While both can bring additional benefits, I only have bandwidth for one at the moment (actually, I don't have bandwidth for either, but I may make room for one).
As for the pass through, we have had that request for HDV and DV in the past. I agree it would be nice, but it is far from trivial for any format, and H.264 is especially difficult. When I consider it against other factors and consider that advanced users will filter a lot (color correction esp. as that improves) and low end users will encode to something for the Internet or DVD, then its usefulness is rather limited. Nevertheless, it is on the MLT todo albeit rather low.
Hi,
just joined because I recently bought a Panasonic HDC-SD300 and I am very interested in editing AVCHD material, too.
Before I had a Mini-DV from Sony which broke down after 9 years of usage now. Kino, Cinelerra and later releases of Kdenlive were exactly what I needed to edit footage with my openSUSE box.
Nevertheless, find it difficult now to edit AVCHD material with any of the available software. Cinelerra doesnt except .mts at all (knowing the workarounds to change to other formats with ffmpeg), so Kdenlive is what almost does the job, even playback, exact frame cut and rendering still is an issue.
Not to get me wrong, I like Kdenlive, because it's very easy to use (much more than Cinelerra) but powerful enough to get my birthday or vacation footage to a nice film. I just wanna dis-agree to the statement that low-end users wanna render in formats like for Youtube or DVD. I can make better videos with my mobile phone to get it to Youtube. When people spend between 600 and 900 Euros for a Camcorder that is able to record in 1080i they usually wanna keep the high quality. Sure, downgrading is still an option to get footage to the internet but in general I wanna get my holidays memory film cut with nice music below playable in 1080i again on my Full-HD TV. And I always kept the raw footage just with some fade in and out, transitions and add musik. I only remember that I had to change the brightness for one clip one time in the past which would result in a complete new rendering.
So imho it should be one of the major focus to get good AVCHD support, even it is difficult. Short rendering time to raw, initial format (AVCHD) would be more than welcome by a lot of people that got a new AVCHD camcorder.
Maybe some PC specs if somebody interested: openSUSE 11.1, KDE 4.2 on an AMD 4600+ (dual core Athlon 2400MHz), 8GB RAM, nvidia 7600GT, 2.5TB HDDs. With this one I rendered 30min AVCHD footage to H264/AC3 5.1 @ 12.000 Kb/s in almost 10 hours.
Cheers,
OsZ
Should it be my major focus even though I do not have an AVCHD camera, am not getting paid for this, and have nothing to gain except, well, purely the glory of being an elite free software hacker? Yeah! Rock On!
IMHO, I should focus on helping Kdenlive be better at formats it does support fairly well. Then, hopefully, someone who really cares about AVCHD and has the skill to do something about might be compelled enough to contribute.
Hi ddennedy, nothing against that and I do not expect anything from open source software that is out of skill of the hackers that feel like you. I even do not know anything about programming otherwise I would help on some projects as well. My only bit to the open source community can be to help testing, and give feedback from a user point of view. And then back to topic: I appreciate that you are maybe not interested because you don't own a new camcorder but in general I always feel the open source hackers are mainly focusing on old stuff. I am system engineer in the automotive industry and we need to look forward when we develop new cars at least 4 years, not backwards whats available. Thats my general thinking in whole life - living for now and the future - past is gone. I am sorry if you do not think so. Again, nothing against you or open source and this great project but a bit forward thinking also in spare time when programming (yes, I know) for free would help a lot for the overall state of open source.
Will use Kdenlive, because thats so far the best open source video editor for daily use. I like the easyness and the way it interacts with me as the user.
For any AVCHD footage samples let me know, for things to test and try out, let me know. I registered my SD300 in the supported camcorders section.
PS: Just before I stop with this kind of conversation, if someday somebody feels like care about AVCHD then Metadata from the corresponding sub-folders should be considered as well, not only the video files (*.mts) themselves.
Thanks a lot...
Cheers,
OsZ
> Should it be my major focus even though I do not have an AVCHD camera, am not
> getting paid for this, and have nothing to gain except, well, purely the glory
> of being an elite free software hacker? Yeah! Rock On!
I appreciate that sentiment, but the majority of new camcorders being sold these days are AVCHD I suspect (or if not, soon will be).
You have the opportunity to be an online hero :) with this OS product that would keep 99% of consumers happy.
It is up to you though.
That's the problem with many Open Source projects - it's great when programmers give away their code that they wrote to "scratch their own itch" but if we want anything else then we're straight outta luck. I had high hopes that kdenlive was the other kind of OS - where the developers are hellbent on the challenge of rivalling the Big Boys :)
Oh well. Good luck with the project anyway. I hope it doesn't meander down a dead-end and go nowhere, because I love the idea of Open Source and love to see succesful projects start from promising beginings like this.
cheers
Actually I'm pretty impressed with the state of AVCHD handling in 0.7.4, and also noted is how much it's improved in a fairly short time. Somebody's doing something right, and it seems to me it can't be that far from being nearly perfected. (Easy for me to say, right?)
Today I edited a pretty short video with about 15 clips. (Canon HF100.) It seems to me that the only blocker I have is in trimming off the front part of a clip, especially if more than a second or so is trimmed. I had one clip where I had to do this, and it played fine on the timeline but that clip wouldn't render right at all. I had to transcode the clip to DNxHD and then import that, then I could trim it. Everything else rendered fine. Clips with substantial trimming off the back end don't seem to have the problem.
How important is to have robust AVCHD handling? Well, check the camera sites and stores. Basically nearly every new camera worth having under $3500 uses it, unless it's a pocket still cam or a Flip. By next year it will basically have taken over the market. If Kdenlive is going to be relevant, well, yeah, it's gonna need to do AVCHD and do it well.
Why don't we just all pitch in and buy Dennedy a new camcorder? :)
I'm expecting the Canon HFS100 to arrive this week (maybe next week). AVCHD is the future (I guess manufacturers choose it long ago).
So that is the conclusion of all this? Everybody will wait until someone else comes along to fix it? Whatever happened to this patch: http://www.kdenlive.org/forum/avchd-playback-and-editing#comment-2524 ?
> Why don't we just all pitch in and buy Dennedy a new camcorder? :)
It still would not solve the problem of Dan not having the bandwidth. I think we should consider it though. Dan is probably our best bet in getting first-class AVCHD support in Kdenlive, as long as the ffmpeg infrastructure needs so much rework to mature.
I do not personally own a AVCHD camera - I explicitely chose a HDV in order to be able to edit that.
However, I would be willing to pitch €100 towards a AVCHD camera for Dan, if he thinks it will improve his motivation for working with AVCHD. Note: one can't ask him to commit to this, only ask if he thinks it will help!
I guess you only lack about 15 others then - if some of you can pledge in some money too, we could make a fundraising thing on e.g. http://www.fundable.com - iff Dan would accept this, of course.
So, what do you say? Are you ready to put your money where your mouth is, or do you just rely for Dan to implement this in due course :-)
Johannes,
MLT can not use that patch because it relies upon FFmpeg to do the demuxing, whereas gmerlin-avdecoder provides its own. There was a guy named Ivan Schreter who submitted a lot of patches towards improving AVCHD in FFmpeg. The comments by ArtInvent reflect that effort. I recently updated FFmpeg to a June snapshot on one of my machines, and I agree it is working much better - usually good enough. However, Ivan's last patch basically overhauls the seeking in FFmpeg and will require updating many of the FFmpeg components (demuxers and decoders). Therefore, it was not accepted yet, and he has lost interest.
I might someday provide a gmerlin-avdecoder backend in MLT. MLT can use multiple backends at the same time. However, first, I have to get it (gmerlin) working and evaluate it. So far, I have been failing. I mean I get something to work, but with very little encoded audio/video support. Needs more time, but I might reject it if I feel it will be more trouble than its worth.
Ultimately, Kdenlive is not the only project interested in AVCHD with accurate seek support in FFmpeg. So, just because I am not willing to work on it at the moment does not mean no one is. Also, buying me a AVCHD camera does mean I can drop everything else related to MLT and Kdenlive just to work on better AVCHD. There are other important bugs and the needs of the distro packagers that I must address as well - not to mention a better deinterlacer. Unfortunately, I am getting little time for much else at the moment. That's really the crux of the problem for me. To address this would require so much reading of the specs and analysis that in order to stay focused on that, I can really only do that.
Hi Dan,
I haven't lost interest, I just got a baby daughter :-). So I don't have time to work on it now. I'm still following the development, though. I've already tried some modifications in MPEG-TS code in order to support seeking properly, however, my efforts so far have failed to achieve workable results, so I didn't post it anywhere.
But I see that Baptiste from FFmpeg group is working now somewhat more intensively on MPEG-TS and H.264 (although in different area), so I'm thinking of getting in touch directly with him.
Regards,
Ivan
@ddennedy: I totally understand you. I am myself a software developer, although in a completely different field and regularly get requests for things that cannot be (easily) done. I just wnated to make sure I had undrstood right what was going on. If I can summarize, it currently looks like this:
- Someone is trying to make VDPAU work in ffmpeg, which would automatically translate into it also working in MLT and kdenlive. This would mean that most people with a modern NVIDIA card would be able to playback and use AVCHD at native speed.
- Someone else might be looking at better software rendering which would include something called PTS/DTS and which would make huge changes to the entire ffmpeg source tree. This should help the rest.
(The presentation of the second point clearly shows I have no clue what I'm talking about.)
Now I was currently looking into buying a new notebook to be able to edit the video, as my current Macbook seems to be too slow for anything. It's one of those using the Intel rather an NVDIA card. But then it hit me: when I run it in Mac OS X and use Final Cut somehow the machine can play the AVCHD footage in the preview window before I even import it into Final Cut (it converts it to Apple's native format during import). First I thought it would just be doing some tricks whereby it would use the camera to play it back, but the same also works if I import from a copied BDMV-folder on an external harddrive.
So I guess my question is this: Is most of the notebook-hardware upgrade we all seem to have to go through now rather pointless in light of it just being the current iteration of ffmpeg/the software that makes it go so slow? Would one expect that once the DTS/PTS stuff goes into ffmpeg, that it will have the same playback quality/speed as the preview of Final Cut on Mac OS X?
from my point of view, it's not a good thing to link fonctionnality and particular hardware ...
nyme
Hi,
for the very brave, my pre-pre-alpha patches for head FFmpeg and head MLT to edit AVCHD files directly in kdenlive (frame-exact seeking support).
Note: current MLT patch will most probably break (at least some of) other formats.
I've tested it with files from my camcorder (Panasonic), other camcorders may have issues (e.g., I know Canon has some issues, so your mileage may vary).
Regards,
Ivan
| Attachment | Size |
|---|---|
| ffmpeg_seek_cooked_lavf.patch | 16.07 KB |
| ffmpeg_seek_cooked_mpegts.patch | 3.87 KB |
| ffmpeg_seek_cooked_seekregress.patch | 4.75 KB |
| mlt_newseek.patch | 9.76 KB |
>> Why don't we just all pitch in and buy Dennedy a new camcorder? :)
> However, I would be willing to pitch €100 towards a AVCHD camera for Dan, if he thinks it will improve
> his motivation for working with AVCHD. Note: one can't ask him to commit to this, only ask if he thinks it will help!
I would also pitch €100 toward such a cause.
Hey and thanks a lot!
The ffmpeg patches compile very cleanly. I had no idea that MLT and kdenlive had switched source repositories though. But when I figured it out also the mlt patch applied cleanly.
When running it looked good at first with seeking working. Playback was still slow, but I guess that's just the processing power. After a few minutes of playing with it it however didn't want to play any more. I instead tried to render just a clip from the timeline to a flv. The resulting flv plays the sound at normal speed but seems to play the video at double speed. That's funny, because it always used to play it at half speed. I wonder if whatever is speeding it up is doing that twice by coincidence.
My camera is a Sony HDR-SR12.
Nice job anyways!
@johanneswilm : mlt framework moved to git at http://mltframework.org/gitweb/mlt.git
do not use svn any more, now the current stable version is mlt 0.4.4 (which includes mlt++ by default).
@sunab: yes I now noticed that. Pretty stupid of me.
@jmpoure ++: Should I post my results as mentioned two posts above somewhere in your database? Should I compile ffmpeg or mlt with debugging enabled and see what makes it crash and post that somewhere? If nobody is going to use it for anything, I won't bother.
Hi all,
since there were no objections from FFmpeg since a week, I've just committed my seeking patches to FFmpeg repository (revision 19681). MPEG-TS (and thus also AVCHD) seeking now works, at least for AVCHD movies from my Panasonic camcorder. I know that Canon camcorders had some issues with proper key frame detection (files are missing appropriate SEI messages), so they might still have a problem.
As for MLT, I've updated my patch for the newest MLT revision (as of today, 2009-08-22). I didn't test it too much, though, but here it is attached. So feel free to test it.
MPEG-PS movies (e.g., DVD VOB) don't work correctly with MLT patch only. Two patches for FFmpeg are still needed, one which introduces proper seeking for MPEG-PS, one to fix a bug in mpegvideo parser. These patches are not yet accepted by FFmpeg, but I attached them here. They also cause seeking regression (patch NOT attached).
Regards,
Ivan
| Attachment | Size |
|---|---|
| producer_avformat.new_.patch | 9.85 KB |
| seek_cooked_mpeg.patch | 4.72 KB |
| mpegvideo_parser_fix.patch | 3.38 KB |
Thanks for publishing these patches.
I've refreshed from the ffmpeg git repository, added the MLT patch, and refreshed my Ubuntu development packages locally. I am doing some local build tests before uploading the new packages to my Ubuntu PPA for anyone that wants to try them out.
See my other post for details of my development snapshot PPA:
http://www.kdenlive.org/forum/development-snapshot-ubuntu-packages
I have a Canon HF11 so I'll provide some feedback for you on my experiences.
The patches cause unacceptably long delays when opening the Canon MTS files - I'm not sure about other camera's files of course.
I have a project with about 20 clips in the project 'tree'. When kdenlive (ffmpeg 0.5+svn20090820, MLT 0.4.4+git20090820 and kdenlive 0.7.5+svn20090818) started it would take about 15-20 seconds to bring up the GUI whilst it scanned the files.
With ffmpeg 4:0.5+svn20090822 (r19681) and MLT 0.4.4+git20090820 + producer_avformat.new_.patch the GUI took more than 10 minutes to appear and most of that time both CPU cores were pushing 100%. Once the GUI was up I selected one MTS clip in the project 'tree' and tried to play it. The clip preview pane remained black whilst the CPU pushed one core to 100% and sat there for several minutes but no playback occured. In the end I closed the program.
I reverted just the MLT packages to the previous version but the long delays and high CPU usage continued so it seems to be the ffmpeg r19681 patches causing this:
cd /var/cache/apt/archives/
sudo dpkg -i libmlt1_0.4.4+git20090820-0ubuntu1~tj~ppa2j_amd64.deb libmlt++2_0.4.4+git20090820-0ubuntu1~tj~ppa2j_amd64.deb melt_0.4.4+git20090820-0ubuntu1~tj~ppa2j_amd64.deb
Is there any useful debugging I can do to help you discover the cause of this?
Hi,
the reason is found: Canon MPEG-TS files don't properly mark key frames, thus seeking reads the whole file. So we need to find some workaround for that. Simplest one is the attached patch, but that won't be accepted in FFmpeg.
Index: libavcodec/h264_parser.c
===================================================================
--- libavcodec/h264_parser.c (revision 19681)
+++ libavcodec/h264_parser.c (working copy)
@@ -168,6 +168,8 @@
/* key frame, since recovery_frame_cnt is set */
s->key_frame = 1;
}
+ if (s->pict_type == FF_I_TYPE)
+ s->key_frame = 1;
pps_id= get_ue_golomb(&h->s.gb);
if(pps_id>=MAX_PPS_COUNT) {
av_log(h->s.avctx, AV_LOG_ERROR, "pps_id out of range\n");
Regards,
Ivan
Using ffmpeg 0.5 release, loading clips and generating thumbnails from my Canon HG10 camera's m2ts files would take several minutes. Using a recent SVN build (from 16 September 2009) it is nearly instant!
You are right, I read that 95% of camcorder market in 2009 would be AVCHD. Impressing, no?