Chrispian.com

Quick vi tip: Show Hidden Characters

2008-01-15

I do a lot of development in linux/unix. In fact, that’s where all my development is, but I spend a fair amount of time in the “CLI” or command line interface. I use putty and ssh in to edit files and look at logs and just generally do what I do.

Today I ran into a problem with a file and I couldn’t figure out what was wrong. I kept getting an error and the spacing was off so I started suspecting a hidden character. vi, the command line editor I use in linux can show you hidden characters. Just type:

1:set list

And bingo, hidden characters are revealed.

Turning them back off is just as easy:

1:set nolist

Write that down, it comes in handy.

Updated 2/25/10: Thanks to Stephen in the comments, I’ve updated the above commands. I had the : in the wrong place!

Greg Bergeson

16 years ago

Thanks for this info on vi. I didn’t want to drag out the books and figure out which option it was again! I really needed this for file conversion between Mac OS and Windows OS (the infamous carriage return versus line feed issue). Thanks again!

Stephen

15 years ago

You don’t need to post my response. I think you meant “:set list” and “:set nolist” instead of “set: list”. I got an error when I typed “set: list” and “:set: list”. Thought yuo might want to know so you can fix your post and save others the headache of figuring it out.

This post was very helpful never the less. Thanks!

Chrispian

15 years ago

Stephen, so right! Totally typo’d that. Updating it now.

Eric

15 years ago

Once you find the characters you want to remove you can use:
:%s///g to remove everything globally. The way to type those pesky special characters in vi (ie ^M) is to use ctrl-v then ctrl-m You then substitute the m for whatever letter you need.

Hope that helps!

Eric

15 years ago

Sorry the posting left out some of my characters. A sample of the above technique would be :%s/^M//g

Sorry for the mixup!

rpp

14 years ago

:%s/.$//g

will take care of you dos file carry over ^M
cheers\r

Boris

14 years ago

Is there a way to show the hidden \t characters?

Thanks,

Mikkel

14 years ago

Also, if you like me would like a key-binding for this, you can use the :set list! version. Notice the “!” at the end. This will toggle list on and off.

Santhi

13 years ago

This is a very useful command. Thanks for sharing this.
We were having troubles running a service using a config file as the config file has hidden characters. I was able to find out the hidden characters in the file using this command and it resolved out issue.
Thanks very much!

Josh Lee

13 years ago

I found out today that :set list in vim doesn’t actually always show dos characters, only sometimes. To make 100% sure you need to run “cat” on the file with the two flags “v” and “T”. So if your file is named foo.bar, you’d type in: “cat -vT foo.bar”

brian

13 years ago

so much easier then getting up and going to my bookshelf to break out the o’reillys book. thanks

Scott

13 years ago

Thank you for the Vi tip. I have been pulling my hair out over some CSS that is not rendering in the browser. Here is the CSS:

.quiz-button-go:hover{
color:#ddffff;
background:#28e07e;
}

Simple right?! The problem is the color is not registering, but the background is. But, when I delete the spaces between the newline and “color”. it works perfectly.

The CSS in question is CSS that I have cut and pasted from my browser into Vim.

When I use :set list, what I get is

44.quiz-button-go:hover{$
45 $
46 ^Icolor:#ddffff;$
47 ^Ibackground:#28e07e;$
48 }$

I’m assuming ^I is tab and $ must be carriage return? Looks fine, right?

Do you know of any other way to view any special hidden chars?

I repeat, if I delete those few spaces, it works perfectly. So I know it’s not a CSS programming thing. This is totally bizarre.

I’ve run into this kind of problem maybe twice in the last 4 months. It’s a real PITA because the text is indeed correct and hunting down this kind of bug is tough…

Bradley Rentfrow

13 years ago

This is what I found to removed the pesky ^M’s that mess up your script:
:e ++ff=dos
:setlocal ff=unix
:w

When you run cat -vT or run the script the ^M’s should be gone.
Basically it saves the file back into the UNIX format and scripts the DOS carriage returns from the file.

guest

4 months ago

Awesome ! thanks .. this is superhelpful

WeatherARC

13 years ago

Thanks for the post. I now just have to decipher what I am seeing. Hmmm running :set list and then seeing some ^I or $ means what?

Deniz Gezmis

12 years ago

If you happen to run Slackware Linux the following command should take care of those DOS related characters smoothly.

fromdos output_file.txt

Michael Monteith

12 years ago

On some systems the vi -b option works great. Here is a good note on the subject
http://www.devdaily.com/blog/post/linux-unix/vi-editor-binary-mode-remove-control-m-characters

JoseMendoza

12 years ago

Thanks for the post, it definitely help to save the reference, since I only use it like once in a while. :-)

Terry Mackie

12 years ago

I was using diff and by eye balling the code, I couldn’t figure out why 2 lines were different. I used set list and saw the 2 lines each had a space at the end. I removed the spaces and diff was sucessful.

Hopping Bunny

11 years ago

Helped me fix a strange problem that cropped up after I added some comments to a shell script !

Go figure.

TD47

11 years ago

Of course, if it is just the pesky ^M from a windows file that is the issue, there is always the good old “dos2unix” converston tool:
dos2unix -o a.xtx b.txt

That will automatically strip out the DOS line feed (carriage return code) stuff. HTH.

Brian S

5 years ago

Another command to find odd and pesky unknown characters is to type “od -cb foobar” where foobar is your file name this will show the file with the character you see and the octal code for that character. The Octal Dump (od) program has many features and options, “man od” for more.

Chrispian

5 years ago

Brian, awesome tip! Thanks for sharing it. I need to compile some of the stuff people have added here to the post.

Paul

5 years ago

for a quick look at special chars, try this

:1,$l

the last character is an “ell” not a “one”

:1,$p prints the standard chars but not special

Mi Sa

5 years ago

This is a great cheat sheet
http://www.lagmonster.org/docs/vi2.html

Chrispian

5 years ago

Mi, that’s awesome! Thanks for sharing.

henry lai

5 years ago

dos2unix -n windows.txt unix.txt

Sal Ferrarello

5 years ago

I was just telling someone about this feature and I’m glad I found this post to share with them – thank you!

Additionally, you can define which hidden characters are displayed and how they appear.

I use Tim Pope’s Vim plugin vim-sensible, which sets the following.

set listchars=tab:>\ ,trail:-,extends:>,precedes:<,nbsp:+

Thanks again for this post!

Paola Simonian

4 years ago

Hi! Thanks a lot for all the info!

I’m still trying to remove the $ that I’ve found… but I’ve tried :%s/$//g and didn’t work. What am I doing wrong?

Thanks!
P

guest

6 months ago

:%s/\n//g

Steve

2 years ago

Wow! 30 years of Unix/Linux and never knew that. My life just got slightly better.

Buddy

2 years ago

Just what I needed a week ago before I rekeyed 4 script files. Now I KNOW I’m not crazy. No worries and and thanks to you that won’t happen again.

Lacey Fow

7 months ago

Thank you for sharing your precious knowledge. Just the right information I needed. By the way, check out my website at QN7 about Cosmetics.

Sam Madrigal

7 months ago

It is always great to come across a page where the admin take an actual effort to generate a really good article. Check out my website YV6 concerning about Web Traffic.