intertwingly

License?

I want to check with mnot first.  Default_ports and the second set of tests are the only substantial reuse from his codebase.  My preference is the Python License.

Hi Sam,

Interesting. Given that you're shooting for RFC2396bis, which gets rid of separate path params, you should probably use urlparse.urlsplit instead of urlparse.urlparse (and likewise urlunsplit instead of urlunparse).

This line:
  (auth,host,port)=re.search('([^@]*@)?([^:]*):?(.*)',auth).groups()
seems to find (userinfo, host, port); is that what you meant?

Similarly what's going on here?
  if auth=="@": auth=""

WRT atom:id, why go through all this when you can just do a lexical compare on the strings; if they're just IDs, why do they need to be normalised at this level?

WRT license, just ack me and link to the original; license yours however you like. I'm also amenable to folding changes back in if you like.

Possibly this is outside of the scope you're imagining for the function, but in the context of normalizing urls 'in the wild', it would be useful to strip() whitespace and line returns from the url as a whole and also perhaps in the parsed fragments as well.  This would resolve cases like:

"http:// www.mysite.com" and

"""http://www.mysite.com

"""

This is going to be difficult since your comment parser will munge these, but here goes:

[link]
[link]
[link]
[link]

Is your intention to produce an ultra-liberal URL normalizer?  If not, you'll need some defensive code to guard against invalid URLs, such as ones that include unescaped high-bit characters.

Wow, that didn't work at all.  Let's try without the scheme.  These are all http URLs:

@example.com/
:@example.com/
127.0.0.1/
127.0.0.1:80/

OK, I've updated urlnorm based on the feedback above.

The way http://:@example.com/ was handled previously was a bug, it now is normalized to http://example.com/.  Unless I'm missing something, http://127.0.0.1/ is correctly normalized.

I'd like non-ASCII characters in URIs to be handled the same way a typical query works: these characters are escaped.

My comment parser won't munge URIs found in code.

I reckon maybe canonicalization isn't such a good idea after all. Too fiddly. I didn't think so before, but after looking at the source, there's an awful lot for someone to get wrong in those 200 loc, and then what if there's any minor change to RFC 2396? Unless someone's prepared to maintain a public domain repository of normalizers for every language under the sun, the additional complexity is asking for non-compliant feeds.

Danny, well then perhaps you shouldn't look at the source.  ;-)

Seriously: can you name one URI on your site that is not normalized?  Actually, looking at your feed, I can name exactly one: http://dannyayers.com, and that particular one is not likely to appear as an entry id.

My plans are to add this code to the feedvalidator - whether this results in error, warning, or informational messages.  So, I'm quite prepared to worry about the "fiddly bits", but I seriously doubt that many other people will have to.

P.S.  Most of the 200 loc are comments, tests, and blank lines.

Should also replace backslashes in the path with slashes:
  r"http://example.com\test.html" --> "http://example.com/test.html"
  r"http://example.com/a\test.html" --> "http://example.com/a/test.html"
  r"http://example.com\a\test.html" --> "http://example.com/a/test.html"

No, backslashes should be translated to %5C.

Looks good so far, but you could add a "bozo" mode to the parser that handles backslashes and unintended spaces as requested above. Just a thought.

This is what I meant when I asked about whether you were planning on making this an ultra-liberal normalizer.  There are lots of goofy things you can do to URLs that work in IE, or particular versions of Netscape, or something.

There are a number of "goofy" things that are legal.  My intent is not to mimic IE, Netscape, etc, but to faithfully implement the rules listed in the initial comment in this source file.  Of course, people are free to compare the output of this function with the input to see if anything changed, and may chose to make value judgments based on this.  In fact, when I originally wrote this function, it was my intent for the feedvalidator to do exactly that.

Suggestions for new rules, comments on the existing rules, and testcases are all welcome.

Hey, this is a nice article! I took the mentioned test cases (and some more) and adjusted them to my own URL "normalizer".
But it is more an URL "fixer" than a normalizer. So RFC compliance is not guaranteed :) For example I replace backslashes with slashes in the path part (as suggested above). This fixes some broken window-ish URL paths.
Implementation (as url_norm()) and unit tests.

[link]...

Hey, Sam. Mind if we use this in iPodder?

Regards,
Garth.

Garth: you are welcome to do so.  My understanding is that the Python license is GPL compatible.

Sam Ruby: Urlnorm by benoit python url Copy | React (0) [link]...

Because this page is a top result for Google searches on the topic, I think it’s worth mentioning some work that has been done on URL normalization with Python that goes beyond urlencoding space characters. For example, dealing with default ports,...

In my environment (Python 2.5.1, urllib 1.17), I had one offending http:// URL:

fer-martin.com/flying-suit-up/

that returned with an exception:

TypeError: decoding Unicode is not supported

I used to pass the URLs to be normalised as: url.encode('utf-8'), so they are <type ‘str'>, but this one kept being <type 'unicode'> even after .encode(). Something was going wrong with the unquote(string) command. I changed clean() as following:

  def clean(string):
  string=unquote(string)
  if type(string) == type(unicode()):
  print 'changing from unicode to str’
  string=string.encode('utf-8')
  string=str(string)
 
  string=unicode(string,'utf-8','replace')
  return unicodedata.normalize('NFC',string).encode('utf-8')

Bandwidth is often one of the first bottlenecks you’ll hit when web crawling. So, it’s in your best interest to crawl each page only once (ignoring recrawls). In order to know that you’ve already crawled a page you need to keep an...

Bandwidth is often one of the first bottlenecks you’ll hit when web crawling. So, it’s in your best interest to crawl each page only once (ignoring recrawls). In order to know that you’ve already crawled a page you need to keep an...

From the very beginning, Davis Cup was founded in 1900 and it was governed by the International Tennis Federation (ITF).

nice one

The Prestigious sporting event in Las Vegas, NV by National Finals Rodeo (NFR) Tickets on Thursday, December 3 2020 with 517 people interested.Get your NFR tickets at Vivid Seats! Find ticket prices, the National Finals Rodeo schedule and seating for this Las Vegas rodeo event.

Tho.. It’s great

St Augustine of Canterbury was the person who probably started the widespread celebration of Christmas in large parts of England by introducing Christianity to the regions run by the Anglo-Saxons in the 6th century other Celtic parts of Britain were already Christian but there aren’t many documents about if or how they celebrated the birth of Jesus.

Well, its a nice post

Well, its a nice post

Keep up the superb work, I read barely any posts on this web site and I consider that your website is genuine fascinating and contains bunches of good data.

Its really a great post by Admin. i enjoyed simply reading all of your post. Simply i wanted to inform you that you have people like me who appreciate your work.

And i wishing you a very very Merry Christmas 2019 and Merry Christmas Wishes for Family in Advanced.

Its really a great post by Admin. i enjoyed simply reading all of your post. Simply i wanted to inform you that you have people like me who appreciate your work.

And i wishing you a very very Merry Christmas 2019 and Merry Christmas Wishes for Family in Advanced.

Its really a great post by Admin. i enjoyed simply reading all of your post. Simply i wanted to inform you that you have people like me who appreciate your work

Its really a great post by Admin. i enjoyed simply reading all of your post. Simply i wanted to inform you that you have people like me who appreciate your work.

Nice post thanks for sharing...

Nice information shared

Hey, do you finding Vijay Mahar Background then today I am giving you 200 Vijay Mahar Background free which you can download these background and your own photo editing, Friends, if you are a Picsart photo editor then all these bk stocks are very profitable for you. You all know how famous photo editing has become in today’s time. All people share social media only by editing their photos. Do not share your photo anywhere without editing the photo Friends, we all know that not all people have a laptop More people keep mobile devices, I have brought this background for all of you so that you can easily edit your photo from your phone.

CB Background-Hello, guys if you are finding Cb background for Photo editing So Don’t Worry today I am providing you 300 CB background Friend Friends, you all also know how famous photo editing has become in today’s time. Nowadays everyone edit their photo and shares it on their social media profile like facebook, WhatsApp, etc. so Today I brought CB editing background for all of you with the help of which you can edit your photo very well if you want to download this file click on the download button and download this file free,

Many photo editors keep searching Picsart background today So today I am telling you about Picsart Background Friend  If you have ever used the Picsart app, then you will know that we edit our photos with the help of this app. Many Android users use this app to do photo editing. Today I am going to give you 300 Picsart background Hd for editing your photo in this app. Friends, you can also download all the background in a zip file. I have given you the link to the zip file so that you will get all the stock HD files. And friends, if you download so many stocks one by one from Google, you will take a lot of time.

Shakira can’t wait to light up the Super Bowl 2020 Live Free stage for one big reason — to celebrate her Latin culture.

This week, the “Hips Don’t Lie” singer, 42, revealed all of the reasons why she is counting down the days until her big performance alongside Jennifer Lopez.

Hello to all, the contents present at this web site are in fact awesome for people experience, well, keep up the
good work fellows.

If you have ever used the Wireless dmx, then you will know that we can buy with the help of this website. Wireless dmx receiver/transmitter transmits standard DMX512 protocol data (generated by console) by wireless way. No time delay when signal data is transmitting, signal data is real time and reliably. This product adopts a 2.4G ISM frequency section. High effective GFSK modulate, communication design; 126 channels jumping frequency automatically, high anti-jamming ability.If you choose to use this system with a DMX optical splitter, you can control an almost unlimited number of DMX channels within your lighting network.

Each December since 1985, DEC. 5-14, 2019 NFR chutes have opened at the Thomas & Mack Center in Las Vegas, located on the University of Nevada, Las Vegas campus.

Thanks for the informative and helpful post, obviously in your blog everything is good . I am Sharing That This coming National Finals Rodeo is a must-watch event if you want to see cowboy ride and participate in bull and bronco activities. The ten-day event is anticipated to turn the atmosphere around the Thomas & Mack Center Las Vegas, starting the 5th to 14th Of December.

That Genuinely is a Exact Invaluable Name for me ,Professional rodeo is very popular around the world, especially in the United States. One of the biggest rodeo events is the annual Wrangler National Finals Rodeo, which will take place in Las Vegas, Nevada from December 5 to 14.

That really is quite a very valuable report for me personally, The National Finals Rodeo is set to take place in Las Vegas, Nevada from December 5th to December 14th. The competition is organized by the rodeos’ major North American governing body, Professional Rodeo Cowboys Association (also known as PRCA). The PRCA organizes the NFR as a championship event to award international titles among 120 contestants competing within several high-difficulty rodeo events.

Anthony Joshua vs Andy Ruiz live stream
Anthony Joshua vs Andy Ruiz Boxing live online
Boxing Tonight Live | MMA live stream

https://joshuavsruizjrlive.com

https://joshuavsruizfight.com

https://boxingtonightlive.com

<A href="https://joshuavsruizjrlive.com">Anthony Joshua vs Andy Ruiz live stream</A>
<A href="https://joshuavsruizfight.com">Anthony Joshua vs Andy Ruiz Boxing live online</A>
<A href="https://boxingtonightlive.com">Boxing Tonight Live | MMA live stream</A>

You can see Dallas Cowboys vs New England Patriots live stream on the internet if you’re enrolled member of FuboTV, the exhibits, films and news. If this game is covered by FuboTV live streaming it’s possible to watch american football match top online TV Channel policy business that has streaming coverage for more than 100+ live sports events, most popular Dallas Cowboys vs New England Patriots NFL Game on your iPhone, iPad, Android or Windows mobile phone. In addition, it is simple to discover video highlights and information in the most well-known sports leagues on the planet

Hey guys if you are getting bored then here FOX TV Watch Penn State vs Ohio State Live STREAM College Football 2019.You can see Penn State vs Ohio State live stream on the internet if you’re enrolled member of FuboTV, the top online TV Channel policy business that has streaming coverage for more than 100+ live sports events, most popular exhibits, films and news. If this game is covered by FuboTV live streaming it’s possible to watch College Football match Penn State vs Ohio State on your iPhone, iPad, Android or Windows mobile phone

Thanks for the helpfull post.Here that You can see Texas A&M vs Georgia live stream on the internet if you’re enrolled member of FuboTV, the top online TV Channel policy business that has streaming coverage for more than 100+ live sports events, most popular exhibits, films and news. If this game is covered by FuboTV live streaming it’s possible to watch College Football match Texas A&M vs Georgia on your iPhone, iPad, Android or Windows mobile phone. In addition, it is simple to discover video highlights and information in the most well-known sports leagues on the planet

It’s an interesting website. The information which you share about The Red thread book is very interesting. I really wish to read this book completely. I surely share this with others.

Anthony Joshua vs Andy Ruiz live stream
Anthony Joshua vs Andy Ruiz Boxing live online
Boxing Tonight Live | MMA live stream

https://joshuavsruizjrlive.com

https://joshuavsruizfight.com

https://boxingtonightlive.com

<A href="https://joshuavsruizjrlive.com">Anthony Joshua vs Andy Ruiz live stream</A>
<A href="https://joshuavsruizfight.com">Anthony Joshua vs Andy Ruiz Boxing live online</A>
<A href="https://boxingtonightlive.com">Boxing Tonight Live | MMA live stream</A>

custom pyramid boxes are printed packaging for your products. custom pyramid packaging is designed with logo and your specification. custom pyramid packaging boxes are available at wholesale rates. custom pyramid box packaging is inspiring packaging solution for your product. Get custom printed pyramid boxes at wholesale rate with free shipping cost all across the  United States and Canada.

TCU vs Texas Tech live

Texas Tech vs TCU Football

TCU Texas Tech Live stream

TCU Horned Frogs vs Texas Tech Red Raiders live

I will say, this information is a first class one. thanks...

When I initially commented I seem to have clicked on the -Notify me when new comments are added- checkbox and from now on each time a comment is added I receive four emails with the same comment. Is there a means you are able to remove me from that service? Kudos!Pinoy Lambingan

You are so interesting! I do not believe I have read through a single thing like this before. So great to discover another person with some original thoughts on this topic. Seriously.. many thanks for starting this up. This site is one thing that is needed on the web, someone with a bit of originality!
Pinoy Tv Channel

costume of kathak dance Nice Blog, thank you so much for sharing this blog.

India’s best CA Final Test Series | CA Inter Test Series | CA Foundation Test Series for Improving your performance in ICAI Exams. catestseries is providing online test series Since 2015. It is featured in various  news websites like Hindustan times and asian news. More than 75000+ students benefited from ca final test series and ca inter test series. CAtestseries succeed to maintain highest positive positive pass percentage in Pan India.

You can avail this great service just for free! You can use NEET medical college predictor 2019 to get a clear idea about your college options

I Like your way of writing the article, can you please publish these kind of more article? You can also check mine for nurses

<a href=”https://bestshoesfornurses.us/”>best shoes for nurses(s)</a>

https://bestshoesfornurses.us

The Canada junior hockey 2020 will take place in Ostrava and Trinec, Czech Republic, from Dec. 26, 2019 through Jan. 5, 2020 2019 WJC

www.office.com/setup - Microsoft Office empowers customers to use the Office  thing on the web and furthermore disengaged by downloading the Office Setup on your contraption. You can download the Office Setup on your device like Windows, Mac and phones by visiting the association www.office.com/setup on the web.
[link]

Wilder Fury rematch online
An action-packed lineup of undercard attractions will enter the ring on Saturday, November 23 at the MGM Grand Garden Arena in Las Vegas Wilder Fury rematch online

NFL Super Bowl 2020 Live stream Online Free in Best Quality Tv broadcast. The 54th Superbowl will be held on 2nd February 2020. Don’t miss the big event.
Super bowl 2020 live

Merry Christmas and Happy New Year 2020

You can see Dallas Cowboys vs New England Patriots live stream on the internet if you’re enrolled member of FuboTV, the exhibits, films and news. If this game is covered by FuboTV live streaming it’s possible to watch american football match top online TV Channel policy business that has streaming coverage for more than 100+ live sports events, most popular Dallas Cowboys vs New England Patriots NFL Game on your iPhone, iPad, Android or Windows mobile phone . تحميل العاب كمبيوتر مجانية

Nice article..

what a great article

Awasome blog.I expanded in an incentive over read your blog area. It is very useful for all society there are generally possible results to suit this blog.NFR room and ticket bundles accessible from Boyd Gaming for the National Finals Rodeo..And please give some more information please visit our webpage you get more information concerning making works.

good bog with nice article.