It’s just data

Urlnorm

urlnorm.py

Passes the tests defined in PaceCanonicalIds.  Passes all but three of the tests defined in MNot's urlnorm.py, as I interpret the specs differently for these three.

Only exercised significantly for http URIs.

Testcases welcome.


License?

Posted by Mark at

I want to check with mnot first.  Default_ports and the second set of tests are the only substantial reuse from his codebase.  My preference is the Python License.

Posted by Sam Ruby at

Hi Sam,

Interesting. Given that you're shooting for RFC2396bis, which gets rid of separate path params, you should probably use urlparse.urlsplit instead of urlparse.urlparse (and likewise urlunsplit instead of urlunparse).

This line:
  (auth,host,port)=re.search('([^@]*@)?([^:]*):?(.*)',auth).groups()
seems to find (userinfo, host, port); is that what you meant?

Similarly what's going on here?
  if auth=="@": auth=""

WRT atom:id, why go through all this when you can just do a lexical compare on the strings; if they're just IDs, why do they need to be normalised at this level?

WRT license, just ack me and link to the original; license yours however you like. I'm also amenable to folding changes back in if you like.

Posted by Mark Nottingham at

Possibly this is outside of the scope you're imagining for the function, but in the context of normalizing urls 'in the wild', it would be useful to strip() whitespace and line returns from the url as a whole and also perhaps in the parsed fragments as well.  This would resolve cases like:

"http:// www.mysite.com" and

"""http://www.mysite.com

"""

Posted by Phil McCluskey at

This is going to be difficult since your comment parser will munge these, but here goes:

[link]
[link]
[link]
[link]

Is your intention to produce an ultra-liberal URL normalizer?  If not, you'll need some defensive code to guard against invalid URLs, such as ones that include unescaped high-bit characters.

Posted by http://diveintomark.org/ at

Wow, that didn't work at all.  Let's try without the scheme.  These are all http URLs:

@example.com/
:@example.com/
127.0.0.1/
127.0.0.1:80/

Posted by Mark at

OK, I've updated urlnorm based on the feedback above.

The way http://:@example.com/ was handled previously was a bug, it now is normalized to http://example.com/.  Unless I'm missing something, http://127.0.0.1/ is correctly normalized.

I'd like non-ASCII characters in URIs to be handled the same way a typical query works: these characters are escaped.

My comment parser won't munge URIs found in code.

Posted by Sam Ruby at

I reckon maybe canonicalization isn't such a good idea after all. Too fiddly. I didn't think so before, but after looking at the source, there's an awful lot for someone to get wrong in those 200 loc, and then what if there's any minor change to RFC 2396? Unless someone's prepared to maintain a public domain repository of normalizers for every language under the sun, the additional complexity is asking for non-compliant feeds.

Posted by Danny at

Danny, well then perhaps you shouldn't look at the source.  ;-)

Seriously: can you name one URI on your site that is not normalized?  Actually, looking at your feed, I can name exactly one: http://dannyayers.com, and that particular one is not likely to appear as an entry id.

My plans are to add this code to the feedvalidator - whether this results in error, warning, or informational messages.  So, I'm quite prepared to worry about the "fiddly bits", but I seriously doubt that many other people will have to.

P.S.  Most of the 200 loc are comments, tests, and blank lines.

Posted by Sam Ruby at

Should also replace backslashes in the path with slashes:
  r"http://example.com\test.html" --> "http://example.com/test.html"
  r"http://example.com/a\test.html" --> "http://example.com/a/test.html"
  r"http://example.com\a\test.html" --> "http://example.com/a/test.html"

Posted by Anonymous at

No, backslashes should be translated to %5C.

Posted by Sam Ruby at

Looks good so far, but you could add a "bozo" mode to the parser that handles backslashes and unintended spaces as requested above. Just a thought.

Posted by Asbjørn Ulsberg at

This is what I meant when I asked about whether you were planning on making this an ultra-liberal normalizer.  There are lots of goofy things you can do to URLs that work in IE, or particular versions of Netscape, or something.

Posted by Mark at

There are a number of "goofy" things that are legal.  My intent is not to mimic IE, Netscape, etc, but to faithfully implement the rules listed in the initial comment in this source file.  Of course, people are free to compare the output of this function with the input to see if anything changed, and may chose to make value judgments based on this.  In fact, when I originally wrote this function, it was my intent for the feedvalidator to do exactly that.

Suggestions for new rules, comments on the existing rules, and testcases are all welcome.

Posted by Sam Ruby at

Hey, this is a nice article! I took the mentioned test cases (and some more) and adjusted them to my own URL "normalizer".
But it is more an URL "fixer" than a normalizer. So RFC compliance is not guaranteed :) For example I replace backslashes with slashes in the path part (as suggested above). This fixes some broken window-ish URL paths.
Implementation (as url_norm()) and unit tests.

Posted by Calvin at

Sam Ruby: Urlnorm

[link]...

Excerpt from del.icio.us/jonas/webstandards at

Hey, Sam. Mind if we use this in iPodder?

Regards,
Garth.

Posted by Garth T Kidd at

Garth: you are welcome to do so.  My understanding is that the Python license is GPL compatible.

Posted by Sam Ruby at

Sam Ruby: Urlnorm

Sam Ruby: Urlnorm by benoit python url Copy | React (0) [link]...

Excerpt from Public marks from user benoit at

Answer by cobra libre for How can I normalize a URL in python

Because this page is a top result for Google searches on the topic, I think it’s worth mentioning some work that has been done on URL normalization with Python that goes beyond urlencoding space characters. For example, dealing with default ports,...

Excerpt from How can I normalize a URL in python - Stack Overflow at

In my environment (Python 2.5.1, urllib 1.17), I had one offending http:// URL:

fer-martin.com/flying-suit-up/

that returned with an exception:

TypeError: decoding Unicode is not supported

I used to pass the URLs to be normalised as: url.encode('utf-8'), so they are <type ‘str'>, but this one kept being <type 'unicode'> even after .encode(). Something was going wrong with the unquote(string) command. I changed clean() as following:

  def clean(string):
  string=unquote(string)
  if type(string) == type(unicode()):
  print 'changing from unicode to str’
  string=string.encode('utf-8')
  string=str(string)
 
  string=unicode(string,'utf-8','replace')
  return unicodedata.normalize('NFC',string).encode('utf-8')

Posted by Manos at

URL Normalization in Clojure

Bandwidth is often one of the first bottlenecks you’ll hit when web crawling. So, it’s in your best interest to crawl each page only once (ignoring recrawls). In order to know that you’ve already crawled a page you need to keep an...

Excerpt from X-Combinator at

URL Normalization in Clojure

Bandwidth is often one of the first bottlenecks you’ll hit when web crawling. So, it’s in your best interest to crawl each page only once (ignoring recrawls). In order to know that you’ve already crawled a page you need to keep an...

Excerpt from Eigenjoy at

From the very beginning, Davis Cup was founded in 1900 and it was governed by the International Tennis Federation (ITF).

Posted by Davis cup finals 2019 stream at

nice one

Posted by kin music at

The Prestigious sporting event in Las Vegas, NV by National Finals Rodeo (NFR) Tickets on Thursday, December 3 2020 with 517 people interested.Get your NFR tickets at Vivid Seats! Find ticket prices, the National Finals Rodeo schedule and seating for this Las Vegas rodeo event.

Posted by NFR 2019 Live Stream at

Tho.. It’s great

Posted by Latest Nigerian Music at

St Augustine of Canterbury was the person who probably started the widespread celebration of Christmas in large parts of England by introducing Christianity to the regions run by the Anglo-Saxons in the 6th century other Celtic parts of Britain were already Christian but there aren’t many documents about if or how they celebrated the birth of Jesus.

Posted by christmas tree angel ornaments at

Well, its a nice post

Posted by download Mp3 Music at

Well, its a nice post

Posted by abdwap at

Keep up the superb work, I read barely any posts on this web site and I consider that your website is genuine fascinating and contains bunches of good data.

Posted by www.medicontips.com at

Its really a great post by Admin. i enjoyed simply reading all of your post. Simply i wanted to inform you that you have people like me who appreciate your work.

And i wishing you a very very Merry Christmas 2019 and Merry Christmas Wishes for Family in Advanced.

Posted by Merry Christmas 2019 at

Its really a great post by Admin. i enjoyed simply reading all of your post. Simply i wanted to inform you that you have people like me who appreciate your work.

And i wishing you a very very Merry Christmas 2019 and Merry Christmas Wishes for Family in Advanced.

Posted by Merry Christmas 2019 at

Its really a great post by Admin. i enjoyed simply reading all of your post. Simply i wanted to inform you that you have people like me who appreciate your work

Posted by pubg pc at

Its really a great post by Admin. i enjoyed simply reading all of your post. Simply i wanted to inform you that you have people like me who appreciate your work.

Posted by Akadstatus at

Nice post thanks for sharing...

Posted by Birthday status for bhabhi at

Nice information shared

Posted by GSIM at

Hey, do you finding Vijay Mahar Background then today I am giving you 200 Vijay Mahar Background free which you can download these background and your own photo editing, Friends, if you are a Picsart photo editor then all these bk stocks are very profitable for you. You all know how famous photo editing has become in today’s time. All people share social media only by editing their photos. Do not share your photo anywhere without editing the photo Friends, we all know that not all people have a laptop More people keep mobile devices, I have brought this background for all of you so that you can easily edit your photo from your phone.

Posted by 200 Top Vijay Mahar Background Download |2020| at

CB Background-Hello, guys if you are finding Cb background for Photo editing So Don’t Worry today I am providing you 300 CB background Friend Friends, you all also know how famous photo editing has become in today’s time. Nowadays everyone edit their photo and shares it on their social media profile like facebook, WhatsApp, etc. so Today I brought CB editing background for all of you with the help of which you can edit your photo very well if you want to download this file click on the download button and download this file free,

Posted by 300 Top CB Background Hd |2019| Zip File at

Many photo editors keep searching Picsart background today So today I am telling you about Picsart Background Friend  If you have ever used the Picsart app, then you will know that we edit our photos with the help of this app. Many Android users use this app to do photo editing. Today I am going to give you 300 Picsart background Hd for editing your photo in this app. Friends, you can also download all the background in a zip file. I have given you the link to the zip file so that you will get all the stock HD files. And friends, if you download so many stocks one by one from Google, you will take a lot of time.

Posted by 300 Picsart Background HD Download|2019| Zip File at

Shakira can’t wait to light up the Super Bowl 2020 Live Free stage for one big reason — to celebrate her Latin culture.

This week, the “Hips Don’t Lie” singer, 42, revealed all of the reasons why she is counting down the days until her big performance alongside Jennifer Lopez.

Posted by Watch Super Bowl at

Hello to all, the contents present at this web site are in fact awesome for people experience, well, keep up the
good work fellows.

Posted by ameer khan at

If you have ever used the Wireless dmx, then you will know that we can buy with the help of this website. Wireless dmx receiver/transmitter transmits standard DMX512 protocol data (generated by console) by wireless way. No time delay when signal data is transmitting, signal data is real time and reliably. This product adopts a 2.4G ISM frequency section. High effective GFSK modulate, communication design; 126 channels jumping frequency automatically, high anti-jamming ability.If you choose to use this system with a DMX optical splitter, you can control an almost unlimited number of DMX channels within your lighting network.

Posted by Sarah at

Each December since 1985, DEC. 5-14, 2019 NFR chutes have opened at the Thomas & Mack Center in Las Vegas, located on the University of Nevada, Las Vegas campus.

Posted by NFR Live Stream Online at

Thanks for the informative and helpful post, obviously in your blog everything is good . I am Sharing That This coming National Finals Rodeo is a must-watch event if you want to see cowboy ride and participate in bull and bronco activities. The ten-day event is anticipated to turn the atmosphere around the Thomas & Mack Center Las Vegas, starting the 5th to 14th Of December.

Posted by Wrangler NFR 2019 at

That Genuinely is a Exact Invaluable Name for me ,Professional rodeo is very popular around the world, especially in the United States. One of the biggest rodeo events is the annual Wrangler National Finals Rodeo, which will take place in Las Vegas, Nevada from December 5 to 14.

Posted by Watch Wrangler NFR live stream at

That really is quite a very valuable report for me personally, The National Finals Rodeo is set to take place in Las Vegas, Nevada from December 5th to December 14th. The competition is organized by the rodeos’ major North American governing body, Professional Rodeo Cowboys Association (also known as PRCA). The PRCA organizes the NFR as a championship event to award international titles among 120 contestants competing within several high-difficulty rodeo events.

Posted by NFR Live Stream at

Anthony Joshua vs Andy Ruiz live stream
Anthony Joshua vs Andy Ruiz Boxing live online
Boxing Tonight Live | MMA live stream

https://joshuavsruizjrlive.com

https://joshuavsruizfight.com

https://boxingtonightlive.com

<A href="https://joshuavsruizjrlive.com">Anthony Joshua vs Andy Ruiz live stream</A>
<A href="https://joshuavsruizfight.com">Anthony Joshua vs Andy Ruiz Boxing live online</A>
<A href="https://boxingtonightlive.com">Boxing Tonight Live | MMA live stream</A>

Posted by joshuavsruizfight at

You can see Dallas Cowboys vs New England Patriots live stream on the internet if you’re enrolled member of FuboTV, the exhibits, films and news. If this game is covered by FuboTV live streaming it’s possible to watch american football match top online TV Channel policy business that has streaming coverage for more than 100+ live sports events, most popular Dallas Cowboys vs New England Patriots NFL Game on your iPhone, iPad, Android or Windows mobile phone. In addition, it is simple to discover video highlights and information in the most well-known sports leagues on the planet

Posted by New England Patriots vs Dallas Cowboys at

Hey guys if you are getting bored then here FOX TV Watch Penn State vs Ohio State Live STREAM College Football 2019.You can see Penn State vs Ohio State live stream on the internet if you’re enrolled member of FuboTV, the top online TV Channel policy business that has streaming coverage for more than 100+ live sports events, most popular exhibits, films and news. If this game is covered by FuboTV live streaming it’s possible to watch College Football match Penn State vs Ohio State on your iPhone, iPad, Android or Windows mobile phone

Posted by Penn State vs Ohio State Live Stream at

Thanks for the helpfull post.Here that You can see Texas A&M vs Georgia live stream on the internet if you’re enrolled member of FuboTV, the top online TV Channel policy business that has streaming coverage for more than 100+ live sports events, most popular exhibits, films and news. If this game is covered by FuboTV live streaming it’s possible to watch College Football match Texas A&M vs Georgia on your iPhone, iPad, Android or Windows mobile phone. In addition, it is simple to discover video highlights and information in the most well-known sports leagues on the planet

Posted by Texas A&M vs Georgia Live Stream at

It’s an interesting website. The information which you share about The Red thread book is very interesting. I really wish to read this book completely. I surely share this with others.

Posted by payday loans no fees no brokers at

Anthony Joshua vs Andy Ruiz live stream
Anthony Joshua vs Andy Ruiz Boxing live online
Boxing Tonight Live | MMA live stream

https://joshuavsruizjrlive.com

https://joshuavsruizfight.com

https://boxingtonightlive.com

<A href="https://joshuavsruizjrlive.com">Anthony Joshua vs Andy Ruiz live stream</A>
<A href="https://joshuavsruizfight.com">Anthony Joshua vs Andy Ruiz Boxing live online</A>
<A href="https://boxingtonightlive.com">Boxing Tonight Live | MMA live stream</A>

Posted by joshuavsruizfight at

custom pyramid boxes are printed packaging for your products. custom pyramid packaging is designed with logo and your specification. custom pyramid packaging boxes are available at wholesale rates. custom pyramid box packaging is inspiring packaging solution for your product. Get custom printed pyramid boxes at wholesale rate with free shipping cost all across the  United States and Canada.

Posted by Buy custom boxes wholesale at

TCU vs Texas Tech live

Texas Tech vs TCU Football

TCU Texas Tech Live stream

TCU Horned Frogs vs Texas Tech Red Raiders live

Posted by Watch college football online at

I will say, this information is a first class one. thanks...

Posted by AbegLoaded at

When I initially commented I seem to have clicked on the -Notify me when new comments are added- checkbox and from now on each time a comment is added I receive four emails with the same comment. Is there a means you are able to remove me from that service? Kudos!Pinoy Lambingan

Posted by Pinoy Tv at

You are so interesting! I do not believe I have read through a single thing like this before. So great to discover another person with some original thoughts on this topic. Seriously.. many thanks for starting this up. This site is one thing that is needed on the web, someone with a bit of originality!
Pinoy Tv Channel

Posted by Pinoy Replay Shows at

costume of kathak dance Nice Blog, thank you so much for sharing this blog.

Posted by kathak costume at

India’s best CA Final Test Series | CA Inter Test Series | CA Foundation Test Series for Improving your performance in ICAI Exams. catestseries is providing online test series Since 2015. It is featured in various  news websites like Hindustan times and asian news. More than 75000+ students benefited from ca final test series and ca inter test series. CAtestseries succeed to maintain highest positive positive pass percentage in Pan India.

Posted by CA Inter Test Series at

You can avail this great service just for free! You can use NEET medical college predictor 2019 to get a clear idea about your college options

Posted by neet college predictor at

I Like your way of writing the article, can you please publish these kind of more article? You can also check mine for nurses

<a href=”https://bestshoesfornurses.us/”>best shoes for nurses(s)</a>

Posted by Saurav Chhabra at

https://bestshoesfornurses.us

Posted by Saurav Chhabra at

The Canada junior hockey 2020 will take place in Ostrava and Trinec, Czech Republic, from Dec. 26, 2019 through Jan. 5, 2020 2019 WJC

Posted by Canada junior hockey 2020 at

www.office.com/setup - Microsoft Office empowers customers to use the Office  thing on the web and furthermore disengaged by downloading the Office Setup on your contraption. You can download the Office Setup on your device like Windows, Mac and phones by visiting the association www.office.com/setup on the web.
[link]

Posted by office.com/setup at

Wilder Fury rematch online
An action-packed lineup of undercard attractions will enter the ring on Saturday, November 23 at the MGM Grand Garden Arena in Las Vegas Wilder Fury rematch online

Posted by Wilder Ortiz 2 full video at

NFL Super Bowl 2020 Live stream Online Free in Best Quality Tv broadcast. The 54th Superbowl will be held on 2nd February 2020. Don’t miss the big event.
Super bowl 2020 live

Posted by Super bowl 2020 live at

Merry Christmas and Happy New Year 2020

Posted by Ahmed Nawaz at

You can see Dallas Cowboys vs New England Patriots live stream on the internet if you’re enrolled member of FuboTV, the exhibits, films and news. If this game is covered by FuboTV live streaming it’s possible to watch american football match top online TV Channel policy business that has streaming coverage for more than 100+ live sports events, most popular Dallas Cowboys vs New England Patriots NFL Game on your iPhone, iPad, Android or Windows mobile phone . تحميل العاب كمبيوتر مجانية

Posted by tota at

Nice article..

Posted by Latest Hip Hop Songs at

what a great article

Posted by Sugar momma App at

Awasome blog.I expanded in an incentive over read your blog area. It is very useful for all society there are generally possible results to suit this blog.NFR room and ticket bundles accessible from Boyd Gaming for the National Finals Rodeo..And please give some more information please visit our webpage you get more information concerning making works.

Posted by Las Vegas Rodeo Tube at

good bog with nice article.

Posted by Sugar Mummy Hookup at

Add your comment